Methods and systems for neural network optimization

Abstract

Methods and systems for optimizing neural networks, and in certain embodiments optimizing neural networks for low power edge devices / sensors and providing fault tolerance for neural network execution. Certain disclosed neural networks can change width during inference, such as using one width for one layer, and using a different width for a later layer and such networks may be trained such that the accuracy at any width / depth combination is equivalent to a standard network (that cannot change) of the same width and depth.

Description

UCT-02225METHODS AND SYSTEMS FOR NEURAL NETWORK OPTIMIZATIONRELATED APPLICATIONS

[0001] This application claims priority to U.S. provisional patent application 63 / 684,116 filed on August 16, 2024, to U.S. provisional patent application 63 / 684,125 filed on August 16, 2024, to U.S. provisional patent application 63 / 767,369 filed on March 5, 2025, and to U.S. provisional patent application 63 / 767,370 filed on March 5, 2025, each of which is hereby incorporated by reference in its entirety.FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

[0002] This invention was made with government support under Grant Nos. CCF-2119184, CNS-1956180, CNS-1952050, and CCF-182303 awarded by the United States National Science Foundation. The government has certain rights in the invention.FIELD OF THE DISCLOSURE

[0003] This disclosure relates to the field of computer-implemented neural networks.BACKGROUND

[0004] Neural network inference models can be characterized by both depth (the number of layers) and width (the size of each layer). Recent progress in intermittent computation and artificial intelligence has enabled deployment of neural network inference models at various points in a network including in low power devices and ultra-low power (ULP) devices at the “edge” of a network.UCT-02225

[0005] Executing deep neural networks (DNNs) on ULP microcontrollers creates enormous opportunities for new intelligent edge applications. However, manually writing optimized DNN programs for ULP devices is time consuming and error prone due to the difficulty of managing on-device accelerators. Many prior works address this problem by creating special libraries that tailor common DNN building blocks for unique accelerators of ULP devices. This is a bottom-up approach, as developers build DNNs by assembling library calls. Unfortunately, the encapsulation overhead inherent in this approach greatly reduces accelerator utilization and overall performance.

[0006] ULP microcontrollers that employ energy-harvesting, intermittent systems enable battery less inference at the edge, but a key challenge is providing accurate inference with bounded latency despite abrupt and frequent energy fluctuations. This has been partially addressed using Adaptive Inference systems that trade inference accuracy for reduced energy. However, these approaches are not robust to the wide range of energy fluctuations that occur at the edge.SUMMARY

[0007] A computer-implemented method for executing a neural -network computation is disclosed herein. The computer-implemented method comprises maintaining a set of control variables that encode progress through the neural network. A checkpoint record is created at continuation points during execution where the checkpoint record comprises values of the control variables and a commit indicator. The checkpoint record is persisted to a non-volatile buffer using an atomic logging scheme in which the commit indicator is written to indicate successful persistence. Upon restart after an interruption, the checkpoint record is read and execution is resumed by restoring the control variables and continuing the neural -network computation from the encoded progress.UCT-02225

[0008] The control variables may correspond to indices of a plurality of nested loops that implement the neural -network computation, including one or more of: layer index, operator index, tensor-tile index, neuron index, or time-step index. The control variables may comprise inter-layer progress and intra-layer progress, the inter-layer progress indicating which layer or operation is active and the intra-layer progress indicating partial completion within the active layer or operation.

[0009] The checkpoint record persistence using the atomic logging scheme may include writing the control variables to the non-volatile buffer and using available hardware support to detect or enforce that the write is committed.

[0010] The commit indicator may be encoded by setting a most-significant bit of at least one control variable as a commit bit, the commit bit being updated only after all control-variable values are durably written to the non-volatile buffer.

[0011] The available hardware support may include transactional memory primitives that provide atomicity of the checkpoint record write, a read-modify -write atomic instruction to update the commit indicator, or load-linked / store-conditional instructions to update the commit indicator conditionally on the absence of conflicting writes.

[0012] A computer-implemented method for performing inference using a neural network model is described herein. The computer-implemented method comprises modifying width of the neural network during inference whereby a first width is employed for a first layer and a second width is employed for a second layer. The computer-implemented method may also include performing an early exit of the neural network whereby one or more layers of the neural network are not employed in generating an output of the inference. One or more inputs provided by an entity external to the neural network may be accepted to select depth and width of the neural network in performing inference.UCT-02225

[0013] A computer-implemented method for training a neural network model characterized by changeable width and depth during inference is disclosed herein. The computer- implemented method comprises shrinking width of the artificial neural network at a rate. A grouping of the rate is characterized by separate batch-norm statistics. The computer- implemented method comprises dropping an entire channel when shrinking width. The computer-implemented method comprises training the artificial neural network.

[0014] A computer program product is disclosed herein. The computer program product comprises a computer readable storage medium having program instructions embodied therewith. The program instructions are executable to perform any methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] So that the manner in which the above recited advantages and features of the present invention, as well as others which will become apparent, are attained and can be understood in detail, a more particular description of the invention summarized above may be had by reference to the embodiments thereof which are illustrated in the appended drawings, which drawings form a part of this specification.

[0016] It is to be noted, however, that the appended drawings illustrate only typical embodiments of the invention and therefore are not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.

[0017] FIG. 1 is a flowchart illustrating operations performed in a computer-implemented method for generating code executable by a digital computer.

[0018] FIG. 2 is a flowchart illustrating further details of an embodiment of FIG. 1 to select a Deep Neural Network (DNN) layer based on measured behavior.

[0019] FIG. 3 is a flowchart illustrating an embodiment of checkpointing to save execution progress of the execution engine of the DNN of FIG. 1.UCT-02225

[0020] FIG. 4 is a flowchart illustrating an embodiment of using available hardware support to indicate status of checkpointing of FIG. 3.

[0021] FIG. 5 is a flowchart illustrating various uses of available hardware support to indicate status of checkpointing of FIG. 3.

[0022] FIG. 6A illustrates an example of pseudo-code for a 2D convolution.

[0023] FIG. 6B is a graph illustrating latencies of LEA functions with different input sizes on a sample processor.

[0024] FIG. 7A and FIG. 7B illustrate two different instantiations of a DNN convolution layer.

[0025] FIG. 8A and FIG. 8B are graphs illustrating convolution performance on a sample processor.

[0026] FIG. 9A is a flow diagram illustrating high-level operation of Lupe which generates code for DNN inferencing.

[0027] FIG. 9B is a flow diagram illustrating further details of the construction operation ofFIG. 9A

[0028] FIG. 10 illustrates an example of pseudo-code to organize instantiations on a 2D convolution.

[0029] FIG. 11 is a graph illustrating latencies for different data movement methods on a sample processor.

[0030] FIG. 12 illustrates an example of pseudo-code to provide inter-layer and intra-layer intermittent support.

[0031] FIG. 13 shows graphs illustrating inference latencies of various models using continuous power.

[0032] FIG. 14 shows graphs illustrating latency per inference, including device initialization time, under different reboot rates.UCT-02225

[0033] FIG. 15 shows graphs illustrating energy consumption of a single DNN inference for both continuous and intermittent-safe implementations.

[0034] FIG. 16A and FIG. 16B show graphs illustrating power snapshots of executing a sample model for respectively, continuous and intermittent-safe implementations.

[0035] FIG. 17 shows a graph illustrating a breakdown of optimizations for Lupe.

[0036] FIG. 18 shows a graph illustrating Low Energy Accelerator (LEA) utilization for Lupe.

[0037] FIG. 19 is a block diagram illustrating an adjustable DNN inferencing engine.

[0038] FIG. 20 is a flow diagram illustrating high-level operation of the disclosed SLUMBER system.

[0039] FIG. 21 is a graph illustrating adaption along width and depth by EarlySlims networks.

[0040] FIG. 22 is a flow diagram illustrating high level operation of the disclosed Progressive Adaptive Control Engine (PACE).

[0041] FIG. 23 shows a graph that illustrates latency of an intermittent inference task.

[0042] FIG. 24 illustrates an example of pseudocode to implement an embodiment of the disclosed PACE.

[0043] FIG. 25 shows a graph that illustrates recharge count underestimation and overestimation for an inference application.

[0044] FIG. 26 shows graphs that illustrate final results of different applications using different adaptive inference approaches.

[0045] FIG. 27 shows graphs that show accuracy and latency of various networks when running on constant power.

[0046] FIG. 28 is a graph illustrating average accuracy under non-volatile power traces.

[0047] FIG. 29 is a graph illustrating experimental results of average application accuracy.UCT-02225

[0048] FIG. 30A and FIG. 30B are graphs illustrating percentage of inferences.

[0049] FIG. 31 is a block diagram of computer hardware that may be employed in certain embodiments of computer systems described herein.

[0050] FIG. 32 is a table of DNN application details used for various experiments.

[0051] FIG. 33 is an exemplary algorithm that may be employed in certain described herein.

[0052] FIG. 34 depicts a computing node, in accordance with one or more embodiments of this disclosure.

[0053] While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawing and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.DETAILED DESCRIPTION

[0054] In the following detailed description, reference will be made to the accompanying drawing(s), in which identical functional elements are designated with like numerals. The aforementioned accompanying drawings show by way of illustration, and not by way of limitation, specific embodiments and implementations consistent with principles of the present invention. These implementations are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other implementations may be utilized and that structural changes and / or substitutions of various elements may be made without departing from the scope and spirit of present invention. The following detailed description is, therefore, not to be construed in a limited sense.UCT-02225

[0055] LUPE

[0056] Disclosed herein are methods and systems for optimizing neural networks, and in certain embodiments optimizing neural networks for low power edge devices / sensors. Also disclosed herein are methods and systems that provide fault tolerance for neural network execution. These innovations can be applied together or separately.

[0057] In one embodiment illustrated in FIG. 1, a computer-implemented method for generating code that is executable by a digital computer is described. The code converts Machine Learning (ML) models encoded in a first format to a second format comprising a Deep Neural Net (DNN) inference library comprising a plurality of layers. In the method, DNN layer construction is performed by selecting a model 102 of the ML models and parsing the selected model by generating layer by layer instantiations 104. For each generated instantiation 106 of the layer instantiations, usage of computation units on the digital computer is optimized by first restructuring the neural network layer 108 to minimize data movement instructions. For each layer instantiation overhead of invoking the computational unit is minimized by batching instructions (or function calls as applicable) 110 to minimize the cost of invoking the computational unit. The computer-implemented method may also include as shown in FIG. 2, selecting 202 a single layer of the DNN 202, generating 204 multiple instantiations 206 for a single layer of the DNN, measuring behavior of each instantiation 208 and selecting a DNN layer based on measured behavior 210. The foregoing method is applicable (but not limited) to a computing system with a central processing unit and one or more hardware linear algebra accelerators where the described techniques would minimize the instructions that move data to the accelerators and then batch individual instructions within the original neural network specification into a set of linear algebra functions to be executed by the accelerators.UCT-02225

[0058] In another embodiment, a computer-implemented method that provides fault tolerance for neural network execution is described. The computer-implemented method executes an inference engine comprising a Deep Neural Net (DNN). In various embodiments, execution progress of the inference engine may be saved by applying loop continuation 302 as a checkpointing mechanism to the entire neural network execution and using atomic logging to preserve the progress state 304. The method may also include implementing layers of the DNN in nested loops 306. Further details of loop continuation are described below in Top- down Intermittent Safety. The DNN may be stateful between layers (inter-layer support) and within any selected layer (intra-layer support) and the inter-layer support and the intra-layer support may be employed to perform loop continuation. The atomic logging may include as shown in FIG. 4 using available hardware support 402 to indicate whether relevant program state has been saved to a permanent buffer 404. The hardware support may comprise one or more of the operations shown in FIG. 5. Operation 502 comprises using the most significant bit of control variables as a commit bit to mark if data is successfully saved 502. Operation 504 comprises using persistent memory support. Operation 506 comprises using read- modify-write support. Operation 508 comprises using load linked and store conditional instructions. When using the most significant bit as a commit bit at operation 502, an atomic write may be performed to track state. This provides a simple mechanism for performing atomic logging. If the hardware doesn’t provide an atomic write, then atomic writes can be effectively performed by performing one or more other instructions.

[0059] Executing deep neural networks (DNNs) on ultra-low-power (ULP) microcontrollers creates enormous opportunities for new intelligent edge applications. However, manually writing optimized DNN programs for ULP devices is time consuming and error prone due to the difficulty of managing on-device accelerators. Many prior works address this problem by creating special libraries that tailor common DNN building blocks for unique accelerators ofUCT-02225ULP devices. This is a bottom-up approach, as developers build DNNs by assembling library calls. Unfortunately, the encapsulation overhead inherent in this approach greatly reduces accelerator utilization and overall performance. Instead, various embodiments disclosed herein may employ a top-down approach. Various embodiments comprise presenting an exemplary embodiment, Lupe, a code generation framework, that converts high-level DNN algorithm descriptions to ULP-optimized code, as shown in FIG. 9 and described in connection therewith. Lupe provides top-down intermittent support that significantly reduces overhead while maintaining intermittent safety. Various embodiments comprise demonstrating Lupe’s benefits on an MSP430, achieving 12.36* and 2.22* average speedup over two prior works across a variety of DNN models in continuous power. Moreover, Lupe reduces the average intermittent runtime costs of prior works by 96.65% and 71.15%, respectively.

[0060] The emergence of small, but powerful, deep neural network (DNN) models enables intelligent applications on ultra-low-power (ULP) devices. Although these devices include on-chip accelerators, they are extremely constrained by small on- chip memories (16-256 KB), 16-bit fixed point formats, and 4-32 MHz CPU frequency. Despite the limited hardware resources, executing these models on device saves tremendous energy — as much as two orders of magnitude — compared to sending raw data back to a cloud or edge server. Additionally, some ULP devices are deployed without batteries and operated intermittently, z.e., they use only power harvested from the environment and stored in small capacitors. Whether power is continuous or intermittent, ULP devices pose challenges to deliver realtime DNN inference. Specifically, efficient DNN execution requires skilled use of the accelerators, but making efficient use of these accelerators requires tedious data manipulation.UCT-02225

[0061] ULP accelerators perform common linear algebra functions with low latency. Prior works provide libraries of DNN components where each library function encapsulates the data manipulation and accelerator calls, abstracting these confusing details away from users. This method may be referred to as a bottom-up approach because users assemble their complete DNN from many calls to individual library components. For example, Tails provides a library of DNN methods by wrapping vendor provided accelerator functions, each of which independently initializes and deallocates the function’s parameters in the accelerator’s dedicated scratchpad memory. This encapsulation is necessary to support the general use case — z.e., each one must manage the scratchpad properly so one function call will not interfere with another (e.g., by leaving the scratchpad or the accelerator in an undefined state) - but this generality creates overhead that lowers accelerator utilization.

[0062] In contrast, the embodiments disclosed herein employ a top-down methodology that starts from the overall DNN structure and maps it directly into accelerated functions, reducing inter-function data manipulation overhead by leveraging the high-level context of the DNN structure. Such a top-down approach has been explored for high-power devices (like GPUs), yet has not integrated such an approach with DNN optimization for ULP devices. Lupe, the code generation framework disclosed herein takes DNN descriptions as input, and outputs optimized, accelerator-integrated code for ULP devices, e.g., the Texas Instruments (TI) MSP430. The advantages of the top-down approach are threefold. First, it opens new opportunities to optimize accelerator related data manipulation holistically. Lupe breaks the bottom -up approach’s inherent encapsulation constraints and organizes the accelerator routines for optimal data movement with minimal redundancy, considering the entire DNN layer structure. Second, once the overhead of data movement is reduced, Lupe’s top-down approach has a greater opportunity to select the most efficient accelerator operation based on high-level DNN structure. Thus, Lupe automatically offloads the burden of pickingUCT-02225 these operations by adapting the accelerator usage to DNN structures. Finally, many ULP systems operate intermittently in a battery-less condition by harvesting energy from the surrounding environment. The disclosed top-down approach allows highly efficient support for intermittent-safe DNN computation. Lupe combines top-down DNN context with the loop continuation (LC) technique to build a low-overhead, intermittent-safe DNN inference system. The disclosed embodiments employ these techniques to provide a code generation framework that converts ML models in ONNX formats to a customized DNN inference library, which is compatible with the MSP430 toolchain. Lupe converts models built from mainstream DNN frameworks-e.g., PyTorch and TensorFlow-to ONNX representations. Moreover, ONNX provides layer fusion operations, such as fusing batch normalizations with convolutions, that can optimize the DNN from the graph level. Lupe then takes the ONNX representation and generates efficient pro-grams that utilize the MSP430’s low-energy accelerator (LEA) for either continuous or intermittent environments while maintaining inference accuracy.

[0063] Various embodiments have compared Lupe with two prior works, Tails and Hawaii, on 5 representative DNN models and 5 datasets. Experimental results show Lupe achieves an average speed up of 12.36x over Tails and 2.22x over Hawaii in continuous conditions. On average, Lupe’s checkpointing schemes reduce intermittent runtime overhead by 96.65% over Tails and 71.15% over Hawaii.

[0064] Benefits and advantages of Lupe include: (i) adopting the top-down approach for DNNs on ULP devices to achieves higher efficiency due to better accelerator utilization; (ii) Lupe’s top-down approach greatly increases accelerator utilization by optimizing away unnecessary data movement; (iii) High accelerator utilization, brought by Lupe’s top-down approach, creates new opportunities for optimization by tailoring the accelerator usage to the specific DNN structure; and (iv) the disclosed lightweight intermittent runtime, aided byUCT-02225Lupe’s atomic logging scheme, that preserves runtime states. With Lupe, ULP device developers can integrate DNN models into application design without manually implementing them, but instead generating accelerated ULP code from high-level DNN descriptions. Lupe greatly improves runtime efficiency in both intermittent and continuous conditions.

[0065] A Top-Down Approach to ULP Devices. The following provides background on implementing DNNs on the MSP430. The following then describes opportunities to reduce data manipulation overhead and discuss how the reduced overhead creates further opportunities for adaptive layer generation. Background on intermittent computing is then provided and a top-down implementation that can reduce the overhead of supporting intermittent safety is described.

[0066] DNN Inference on the MSP430. The TI MSP430 processors are widely used in ULP computing. MSP430 devices contain 2 compute components: a CPU and a low-energy accelerator (LEA); each of these has a small SRAM scratchpad. For larger storage, the device has a non-volatile memory, FRAM. Finally, a direct memory access (DMA) unit can efficiently transfer data between SRAMs and FRAM.

[0067] The LEA implements efficient linear algebra and signal processing functions. To use an LEA function, one must move data from FRAM to the LEA’s SRAM, allocate function parameters on the LEA’s stack (also located on the scratchpad), and then invoke the LEA function from a C program. After the LEA function completes, these parameters must be freed. Additionally, the LEA manages an internal cache and when LEA functions are executed the LEA automatically copies some data from its SRAM to its internal cache, which creates some additional overhead.

[0068] There are often multiple ways to assemble DNN structures from LEA functions. For example, the LEA’s finite-impulse response (FIR) filter and vector addition may be used toUCT-02225 assemble 2D convolutions (the FIR is essentially a ID convolution). Alternatively, the multiply and accumulate (MAC) operation that serves as a vector dot product may also be used.

[0069] Unfortunately, the LEA’s tremendous performance improvements come with a cost. The required data movement and SRAM stack manipulation are tedious and error prone. Thus, the vendor packages LEA functions into special libraries to abstract away these tricky details. This is an example of the bottom-up approach, and breaking this approach’s encapsulation introduces new optimization opportunities.

[0070] Reducing Data Manipulation Overhead. FIG. 6A is a bottom-up instantiation, or implementation, of a 2D convolution (common in DNN models) where an LEA function consists of three stages: SRAM preparation, invocation and compute. FIG. 6B depicts latencies of LEA functions with different input sizes on an MSP430FR5994 with a 16 MHz CPU and an 8 MHz FRAM. Preparation time, invocation time and compute time are shown from top to bottom for each shown input size on the horizontal scale. When input sizes are small, the overhead of preparation and invocation is relatively higher. This instantiation uses the vendor-supplied LEA FIR and vector addition functions in the innermost loop. Every LEA function can be divided into three parts, namely, SRAM preparation, invocation2, and compute. SRAM preparation moves data to the scratchpad and manages the LEA stack, as described earlier. Various embodiments comprise using linear regression to estimate the invocation time as the real invocation time is hard to measure. The msp lea invoke Command triggers an invocation stage inside LEA, that copies data from its SRAM to its internal cache. Compute refers to time spent purely on computation. FIG. 6A shows that much of SRAM preparation is redundant within a DNN layer because this instantiation repeatedly calls the same LEA functions with the same sizes. FIG. 6B shows the latencies of these three parts while increasing the input data size. Preparation and invocation are dominantUCT-02225 when input sizes are small; the TinyML models used in various embodiments operate on relatively small sizes (vector lengths of 52 or less). Here, it can be recognized that a top-down approach can (1) minimize data manipulation costs by decoupling unnecessary SRAM preparation from actual computation and (2) increase LEA function input sizes to reduce invocation overhead for every instantiation.

[0071] In FIG. 6A, an SRAM preparation stage dynamically allocates a data block on the scratchpad and initializes it with the operation’s parameters. When computation finishes, it deallocates the data block. Although the data block needs to be properly set up, various embodiments comprise reorganizing the program to move it outside LEA functions to minimize preparation costs. Specifically, certain embodiments comprise allocating and / or initializing the parameters at the beginning of DNN layer instantiations and deallocate them in the end. Therefore, most of the preparation costs are eliminated. Unlike the scratchpad, programmers have no control over the LEA’s internal cache. However, various embodiments comprise using batched acceleration (z.e., replace many small FIR calls with a single FIR call to a much larger vector) to amortize the invocation overhead. Thus, by using the high-level DNN structure, the top-down approach provides a systematic way to reduce data manipulation costs for both preparation and invocation stages.

[0072] Adaptive DNN Layer Generation. As LEA supports multiple linear algebra and signal processing functions, there are multiple ways to implement a DNN layer using LEA. Making the most efficient use of LEA requires selecting the best function for a layer.

[0073] For example, FIGS. 7 A and 7B illustrate two different instantiations of a DNN convolution layer for a single input / output channel using a k - k kernel with stride of 1, where one uses the FIR function and the other uses MAC. FIG. 7A illustrates computation of a 2D convolution using the hardware-accelerated FIR and (vector) ADD functions. FIG. 7B illustrates computation of a 2D convolution using the hardware-accelerated MAC function.UCT-02225The FIR-instantiation breaks the input into n blocks, n = 6 in the example, corresponding to the height of the output, as shown in FIG. 7A. The FIR, i.e. ID convolution, may be applied k times on each row within the block and k rows of results may be accumulated using vector addition. The MAC-instantiation, in Figure 7B, breaks the input into n * n blocks. A k x k MAC may be applied on every block to get the output elements.

[0074] Together, FIGS. 7A and 7B illustrate how to compute a 2D convolution on a k x k kernel using the hardware-accelerated FIR and MAC respectively. To use FIR, various embodiments comprise applying it to k input rows separately and use vector addition to aggregate FIR results for every output row. For MAC-instantiations, various embodiments comprise using a k x k MAC and write the results to output buffers directly. Although either approach can be used to compute convolutions, they perform differently based on convolution’s sizes.

[0075] Unfortunately, the overhead of SRAM preparation and invocation makes it hard to realize gains from tuning DNN implementations to the right LEA function. In fact, given the overhead, the best layer instantiation is typically the one that makes the fewest LEA calls. The FIR-instantiation calls LEA functions 2 x n x k times, while the MAC-instantiation calls them n x n times. Given that n » k, FIR instantiations will outperform MAC-instantiations in most cases. Various embodiments comprise benchmarking the performance of these two instantiations on a variety of DNN settings, as shown in FIGS. 8 A and 8B which together show convolution performance of 3 * 3 kernels (FIG. 8A) without, and (FIG. 8B) with data manipulation optimization, both using an MSP430FR5994 with a 16 MHz CPU and an 8 MHz FRAM. The boundary where one implementation would be favored is highlighted with solid lines 802 and 803. In FIG. 8 A. the region 804 (above line 802) reflects FIR instantiations and the region 805 (below line 802) reflects MAC instantiations. In FIG. 8B. the region 806 (above line 803) reflects FIR instantiations and the region 807 (below lineUCT-02225803) reflects MAC instantiations. Darker shades represent higher performance. With the default data manipulation (FIG. 6), FIR is almost always faster. With optimal data manipulation as described below in connection with DNN construction there is a richer tradeoff space.

[0076] As seen in FIG. 8A, without minimizing data manipulation, the MAC-instantiations are rarely higher in performance. Various embodiments comprise reducing data manipulation.

[0077] It can be observed that a top-down approach creates a richer optimization space for adapting DNN instantiations to accelerated functions. Once the top-down optimization demonstrated in connection with the description accompanying FIGS. 6A and 6B is enabled, neither of the two instantiations have a dominant performance advantage across all data sizes as shown in FIG. 8B (note that the solid line denotes the boundary where one instantiation is faster than another). Specifically, FIR-instantiations tend to perform better for small numbers of channels and large input sizes, while MAC-instantiations perform better for large number of channels and small input sizes. A top-down approach can adapt the layer instantiations to the most efficient accelerator functions.

[0078] Intermittent-Safe Computation. By harvesting energy from the environment, e.g. solar energy, radio frequency, etc., intermittent systems can be battery -free and self- sustainable. When energy is sufficient, the system executes programs as normal. As energy is depleted, it preserves runtime states, recharges, and then restores states to continue execution. A major challenge of intermittent systems is to guarantee computation’s correctness given unpredictable power outages. One approach is checkpointing, i.e., automatically recording and restoring runtime states. Checkpointing can have high overhead due to the need to save and restore states. An alternative approach is to use a task system such that the entire program is divided into multiple, atomic tasks and progress is automatically saved in between tasks.Tasks are atomic — they either execute completely or not at all — so they need to be short codeUCT-02225 blocks. Since task-based systems encourage thinking about the program as small blocks of atomic code, they naturally encourage a bottom-up style that obscures high-level, algorithmic information. And while they may be lower-overhead than checkpointing there is still overhead of saving states at each task transition.

[0079] Top-down Intermittent Safety. The Tails system for intermittent-safe DNN execution introduces the idea of a Loop Continuation (LC) for intermittent safety. The idea is to create longer-running tasks with loops (such as a 2D convolution) but use LC as a lightweight “checkpoint” within a loop-based task to reduce task maintenance overhead. LC works under the assumption that the program control flow-even under intermittent power disruptions-is deterministic if the current output and loop indices are preserved across power outages. This approach is thus lower overhead than either checkpointing or tasking alone. However, the Tails system uses a bottom-up programming approach, replacing some tasks with LC. While this reduces task maintenance time, the Tails evaluation still found that approximately 60% of total execution time is spent on task maintenance. This overhead can be further reduced by applying a top-down approach to LC. Here it can be appreciated in view of the present disclosure that a top-down approach can apply the key idea behind loop continuation — a lightweight checkpoint preserving the current output and loop indices — to an entire DNN program to avoid the overhead of task maintenance during DNN inference. Specifically, various embodiments comprise treating the entire DNN program as a single task. Of course, to use LC for the whole DNN program, various embodiments may need to atomically log multiple data words without causing data corruption. The discussion below about Checkpoint Insertion describes how Lupe achieves this atomic logging.

[0080] Lupe Code Generation Framework Design. Various embodiments comprise implementing Lupe as a code generation framework for both continuous and intermittent-safe DNN inferences. FIG. 9 A shows Lupe’s internal workflow and the system design of LupeUCT-02225 which takes a DNN model 901 in the ONNX format and generates an optimized program 902 for a microcontroller 903 such as MSP430FR5994. Lupe takes an input DNN model 901 in the ONNX format and generates programs 902, for either continuous or intermittent computation, that can be deployed on a microcontroller 901 such as a MSP430FR5994. The code generation comprises at least three phases: Construction 904, Calibration 905, and Checkpoint Insertion 906. The construction phase 904 works on each DNN layer sequentially. It first reorganizes programs to minimize SRAM preparation and LEA invocation overhead (as described above - Reducing Data Manipulation Overhead). It then generates multiple instantiations (shown generally at 908) for each DNN layer using different LEA functions (for example, generating both FIR and MAC instantiations of convolutional layers, as described above - Adaptive DNN Layer Generation). The calibration step 905 runs these different instantiations and times 910 them to select the best performing one 912, completing the process of adaptive layer generation. Finally, the checkpoint insertion phase 906 inserts loop continuations as described in Top-Down Intermittent Safety.

[0081] DNN Layer Construction. In the DNN Layer Construction phase shown in further detail in FIG. 9B, Lupe parses the given DNN model 901 and generates instantiations (LEA accelerated implementations) layer by layer 908. Multiple instantiations for a single layer may be generated (and final layer will be picked based on calibration). For each generated instantiation, Lupe attempts to minimize the cost of accelerator usage 920 as motivated in Reducing Data Manipulation Overhead. Specifically, it first generates code to minimize data preparation costs (see Reorganize Programs to Eliminate Preparation Costs) and then batches accelerator function calls to reduce accelerator invocation overhead (see Batch Acceleration to Reduce Invocation Costs).

[0082] Reorganize Programs to Eliminate Preparation Costs. Every DNN layer that uses LEA can be reorganized to eliminate preparation costs. Various embodiments comprise usingUCT-02225 convolutional layers as an example of how to systematically lower these costs. The same ideas, however, apply to a wide range of DNN layer types. The same ideas, however, apply to a wide range of DNN layer types. FIG. 7A shows how to compute a 2D convolution layer 702 using the LEA’s FIR functions. The implementation consists of four levels of for loop, where the LEA FIR function, i.e. msp_fir_ql5, is called row by row. FIG. 6A shows the internals of this function to demonstrate how it interacts with the LEA and the LEA scratchpad. The FIR function allocates its parameters on the LEA stack and initializes them with the appropriate sizes (as specified in the function call). Then, it triggers LEA interrupts to invoke LEA execution. When execution finishes, it deallocates the parameters so the LEA can be used by another function.

[0083] Within a convolutional layer, many of the FIR’s parameters are redundant — e.g., the vector and kernel size — and those parameters occupy a fixed location in the LEA SRAM, even across multiple FIR invocations. Therefore, various embodiments comprise moving the allocation and deallocation of these repeated FIR parameters to the beginning and the end of the convolutional layer, respectively. As convolutions are broken into blocks, that take the same input and kernel sizes, Lupe only needs to change the data addresses for the input, output, and kernel weights. Therefore, only actual LEA computation of FIR is called repeatedly. Figure 10 illustrates Lupe’s reorganization of the FIR-instantiation and specifically how to reorganize instantiations on a 2D convolution using FIR functions. Various embodiments comprise reorganizing programs such that statements in SRAM preparation are called as few as possible. Most of them only need to be called once, at the beginning and end of the instantiation.

[0084] Although Lupe minimizes changes to FIR parameters, the vector addition parameters may need to be modified. Specifically, Lupe uses vector addition on LEA to accumulate FIR results and compute bias offsets, where the input sizes for addition are different and need toUCT-02225 be updated. In general, Lupe allocates parameters of LEA functions on SRAM when entering the DNN layer and does not deallocate them until exiting this layer. Various embodiments comprise initializing operational sizes of LEA functions properly. These sizes may be changed when LEA functions are reused in the same instantiations for different purposes. Before every LEA execution, various embodiments may need to update the addresses of related data, which is moved to the scratchpad before execution.

[0085] Batch Acceleration to Reduce Invocation Costs. Lupe uses the LEA’s FIR and MAC functions to instantiate both normal 2D convolutions as well and depth wise separable ones. Both instantiations are explained in FIG. 7 A and 7B. The code must pay the invocation cost whenever an LEA computation is invoked. Therefore, to further improve LEA utilization, Lupe introduces batched-FIR-instantiations and batched-MAC -instantiations for convolutions which occupies more than 95% of the total execution time. Various embodiments comprise redesigning the entire instantiations to batch data, thereby reducing invocation costs of all LEA functions in these instantiations.

[0086] As described above in connection with the discussion of Adaptive DNN Layer Generation, FIR applies the same kernel to each block row by row. Batched-FIR, on the other hand, concatenates multiple input rows that correspond to the same kernel row together and calls a single FIR on the concatenated row. Batched-FIR works better than batched-MAC when the input sizes are big because the entire LEA’s SRAM can be filled.

[0087] Batched-MAC concatenates input blocks and kernels across multiple input channels and does a MAC in one function call. It works well when there are a large number of channels so that the whole SRAM can be used. However, neither MAC nor batched-MAC is preferred under large input sizes because MAC operations need to reshape the input tensor and reshaping, that can be only done through CPU, is not efficient on MSP430.UCT-02225

[0088] Both batched operations try to enlarge data fed into LEA as much as possible so that Lupe produces the fewest possible LEA calls. The only limit on batch sizes is the amount of data that Lupe can fit into the LEA SRAM. Additionally, batched operations also help improve overall performance of DMA as data with larger sizes is transferred.

[0089] Generate Efficient Data Movement. Lupe uses two methods to manage data movement, DMA calls and unrolled loop copy. Various embodiments comprise optimizing DMA calls from a top-down view based on the same insight of optimizing LEA function calls. Various embodiments comprise selecting which method to use according to statistics performance and DNN configurations.

[0090] A DMA call requires users to set DMA registers for input / out- put addresses, DMA transfer sizes, and DMA channels, and enable the DMA transfer by signaling the DMA control register. Each DMA channel can be configured independently. Lupe only uses one DMA channel for the entire program. The device vendor, i.e. TI, provides packaged DMA calls to handle all such tedious, boilerplate code. However, such a bottom-up approach introduces extra data manipulation overhead (of these registers). Specifically, the register offsets for different DMA channels are calculated dynamically, although register offsets are fixed, because general vendor functions make no assumption on which channel is used. Lupe, on the other hand, forces program to use the same DMA channels, i.e. channel 0. Therefore, Lupe can directly write values to these registers, diminishing the register offset calculation costs.

[0091] Although DMA is fast, its register preparation, or configuration, costs may exceed data movement costs when sizes are small, which is true for the last few layers of some DNN models. Therefore, Lupe uses unrolled loop copy instead, when faster. FIG. 11 shows latencies for different data movement methods on MSP430FR5994. In FIG. 11, data movement by DMA is shown at 1102, loop copy (Unrolled) is shown at 1103, memcpy isUCT-02225 shown at 1104 and Loop Copy is shown at 1105. An unrolled loop copy is used for small sizes and DMA copy for other cases. Particularly, Lupe uses unrolled loop copy for word sizes less than 10 as it has additional costs for keeping track of the array indices.

[0092] Summary of Lupe ’s DNN Instantiations. Table 1 lists instantiations of supported DNN layers in Lupe and specifically, acceleration methods for different DNN instantiations.

[0093] Table 1

[0094] Lupe provides two instantiations for 2D convolutions, including depthwise and pointwise convolutions, and fully connected layers. Shortcut connections have a single instantiation. Pooling and activations, computed on CPU, also have a single instantiation. . Each instantiation is denoted by the main LEA function it uses. Many other LEA functions are also used; e.g., Lupe uses msp_deinterleave_ql5 to extract the stride results for convolutions with stride sizes greater than one. All LEA functions are reorganized to reduce preparation costs as described above in Reorganize Programs to Eliminate Preparation Costs.

[0095] Calibrating Layer Instantiations. Based on the insight from the discussion on Adaptive DNN Layer Generation, Lupe generates DNN layers adaptively. Various embodiments described herein implement part of that idea: Lupe will generate multiple candidate instantiations for a layer, some using FIR and some using MAC, for example. The final step of adaptive generation, then, is to select the best performing instantiation for a particular layer.UCT-02225

[0096] Select Best-Performing Accelerations. Lupe times each generated instantiation individually on the MSP430, from the Construction phase, listed in Table 1, to select the most efficient one in every layer. Performance of DNN inferences on the device is stable, less than 0.001% fluctuation, so real-time benchmarking can better reflect the layer performance comparing to static analysis. A Lupe generated DNN program consists of sequential calls of DNN layers with no parallelization between DNN layers. Therefore, an optimal implementation of every layer (in this program) leads to the best-performing DNN program. Lupe aggregates the fastest instantiation for each layer to build the entire model. Particularly, Lupe can be easily extended to newly added instantiations when new acceleration methods are available.

[0097] Checkpoint Insertion. To ensure correctness of DNN programs in intermittent environments, Lupe must save execution results without data corruption and recover runtime states after power outages. As discussed above in Top-Down Intermittent Safety, Lupe uses loop continuation (LC) with atomic logging to achieve intermittent safety while using a single task for an entire DNN program, eliminating task maintenance during inference. Lupe augments continuous programs from the Calibration phase with support for intermittent safety. The below section on Preserve DNN States Seamlessly describes how Lupe applies LC to the whole DNN program, the section on Log Intermittent Information Atomically explains how atomic logging works.

[0098] Preserve DNN States Seamlessly. Lupe relies on two variations of LC, a loop-LC and a switch-LC, differentiated by types of control flows. Control flows of these two variations are deterministic if the control variables (CVs) are properly saved. Loop-LC constitutes the backbone of Lupe’ s DNN programs as it implements DNN layers in nested loops. Lupe’s implementation of intermittent DNN programs are stateful for both inter-layer and intra-layer support. Therefore, a switch-LC helps automatically preserve program stages. VariousUCT-02225 embodiments comprise illustrating Lupe’s inter-layer and intralayer intermittent support in FIG. 12. Loop-LC and switch-LC may be used to recover runtime states after power outages, while atomic logging saves information atomically avoiding data corruption. Atomic logging may be implemented by using the MSB of CVs as a commit bit to mark if data is successfully saved to a temporary buffer.

[0099] For inter-layer support, Lupe uses a switch-LC to record the DNN inference progress assuming both execution results and run- time states are well-preserved within DNN layers.

[0100] Within layers, Lupe uses a mix of loop-LC and switch-LC. DNN instantiations consist of four stages: (i) preprocess, in which inputs are preprocessed for padding and set outputs to zeros if needed, (ii) main, in which the computation is executed, (iii) bias, in which bias offsets are added, and (iv) exit, in which CVs for the next layer are reset. A recovery function is used to restore the runtime states, e.g. CVs and execution results, when power outages happen during inference.

[0101] Logging multiple pieces of information atomically is an essential component of Lupe’s intermittent runtime, whereas MSP430 has no hardware support for it. Therefore, various embodiments comprise implementing a software atomic logging scheme, i.e. atomic logging, that saves CVs and execution results atomically, explained in the section on Log Intermittent Information Atomically Section.

[0102] Lupe’s LC scheme obviates the need for a task system during DNN inference and greatly reduce the intermittent runtime over- head. Importantly, LC guarantees correctness of runtime only if information is logged atomically.

[0103] Log Intermittent Information Atomically . Atomic logging is necessary to prevent data corruption. While Tails does it through a task system, various embodiments provide a much cheaper solution. Lupe’s atomic logging scheme is built on four assumptions: (i) Writing single variables to non-volatile memory is an atomic, operation, that is guaranteed by theUCT-02225 hardware, (ii) Instructions are executed in-order, that is preserved intact by the hardware, (iii) Only one CV is changed from one control flow block to the next. This is true as the control flow in Lupe instantiations is continuous, namely that Lupe does not use loop-break or goto paradigms. This may be achieved by simply preventing Lupe from generating such code, (iv) Values of CVs will not exceed 215 because the most significant bit, MSB, of CVs is preserved for a commit bit. This commit bit is the key insight for achieving atomic logging. Fortunately, this upper bound is far greater than any CV values observed. The implementation has less than a thousand states, i.e. switch-CVs, in the entire DNN program and loop CVs are used to record input / output channels, rows / columns, that are smaller than 100.

[0104] The first two assumptions enforce that Lupe’s atomic logging scheme will always be executed in the correct order. The third assumption guarantees that the tuple of saved CVs has a one-to-one mapping to control flow blocks, ergo data addresses. The last assumption ensures that Lupe will never corrupt the commit bit.

[0105] In FIG. 12, the atomic logging function provides an exemplary implantation for logging multiple pieces of information atomically. In this example, the goal is to save data from in to out and write val to a CV, st. The data from in may first be written to a temporary buffer. Then, the commit bit and st may be updated simultaneously by using the MSB of st as the commit bit to indicate data is successfully saved. Finally, the data may be written to out and the commit bit cleared sequentially.

[0106] During recovery, if the commit bit is set, runtime states may be recovered using CVs and data may be copied to out from the temporary buffer. The commit bit is cleared after the recovery. On the contrary, if the commit bit is not set, which means either data was just finished being saved from in to out or the data is not fully saved to the temporary buffer yet. In both cases, no actions may be taken.UCT-02225

[0107] The benefit of having atomic logging is twofold. First, it preserves runtime information with little data movement. Second, it only demands a small buffer to hold the temporary data, namely the particular row being operated on. In this case, at most a 4KB buffer may be needed to hold the LEA’s SRAM. For a simple FIR-instantiation, intermittent safety may be achieved with only 80 additional bytes for all benchmarked models. The largest output width in the benchmarks is 32 and 8 CVs may be needed in total.

[0108] With the LC scheme and atomic logging, the data manipulation overhead of preserving intermittent runtime may be greatly reduced.

[0109] Evaluation Setup. Modifications to relevant implementations are discussed below in the section entitled Points of Comparison. Hardware setup and benchmarked models are described in the section entitled Evaluation Methodology.

[0110] Points of Comparison. Various embodiments comprise comparing Lupe generated programs with three relevant implementations under both continuous and intermittent power. Particularly, no intermittent runtime is included when measuring with continuous power. [OHl] Tails is the backend of the LEA-accelerated DNN library for a task-based intermittent-safe computing framework (its CPU-counterpart is called Sonic). Tails implements all convolutions by FIR-instantiations from a bottom-up view, using a task system with loop continuation, to guarantee atomic information updates.

[0112] Hawaii is an intermittent inference library with limited DNN support. It is designed primarily to produce low-overhead intermittent safety over general efficient DNN execution. Hawaii is relevant for comparison since its support for intermittent safety uses a top-down approach, but it does not appear to explicitly apply other top-down optimization opportunities (such as minimizing data movement, batching accelerator calls, and adaptively generating layers). More specifically, Hawaii reduces intermittent safety overhead by preserving SRAM state and thus tries to use the LEA SRAM as much as possible. This design indicates thatUCT-02225Hawaii achieves some of the benefits of the top-down approach (high SRAM utilization) as a side effect.

[0113] Notably, Hawaii also uses LC for intermittent support. However, it does not add support for atomic logging and therefore requires a very large buffer to hold temporary data across multiple input rows to recover data after power outages. In addition, Hawaii does not support convolutions with padding or strides greater than 1. Nor does it support depthwise convolutions and shortcut connections. Moreover, Hawaii assumes that for one channel, both entire input and output can fit into the LEA’s 4k SRAM, which is an invalid assumption except for the smallest model, MLPClassifier. Given these limitations, to benchmark Hawaii, the computation may be divided into multiple chunks while using the entire SRAM. Still, fitting MobileNetV2 into the on-chip FRAM for Hawaii fails as it requires a larger buffer than can fit. All missing modules (e.g., shortcut connection, etc.) may be added to Hawaii from Lupe to have a fair comparison.

[0114] Lupe-BT is implemented from a bottom-up view but adds intermittent runtime using one or more LC schemes. Similar to Tails, Lupe-BT uses FIR-instantiations for all convolutions and MAC -instantiations for all fully connected layers. Lupe-BT ought to have performance comparable to Tails in continuous power as both simply wrap vendor supplied libraries and operate in a bottom up approach.

[0115] Evaluation Methodology. Various embodiments comprise evaluating Lupe and prior works on an MSP430FR5994 with its LEA and DMA enabled. All experiments are conducted under 16MHz CPU frequency and 8MHz FRAM with -02 optimization. A software rebooter, based on the timer, is implemented to simulate intermittent behavior. The software rebooter will trigger an interrupt and restart the device when it reaches the set time. The model function may be wrapped with the software rebooter.UCT-02225

[0116] Performance may be evaluated on a variety of models and datasets. The model architectures in the benchmarks can be found in Table 2.

[0117] Table 2Model Architecture Dataset AccuracyFLOPs Input Size Exp. / Act.Conv 10 x 3 x 3 x 3ResNet3 ® (Channels: 10, Stride: 1) CIFAR-10 80.42%B (Channels: 20, Stride: 2)5.256 M B (Channels: 40, Stride: 2) 3 X 32 X 32 80.48%FC 10 x 40Conv 64 x 1 x 3 x 3 (Stride:2)DS-CNN DS (Channels: 64, Stride: 1) SC 94.39%DS (Channels: 64, Stride: 1)3.049 M DS (Channels: 64, Stride: 1) 1 x 49 x 12 95.18%DS (Channels: 64, Stride: 1)FC 11 x 6477? (Channels: 16, Stride: 2)77? (Channels: 24, Stride: 2)MobileNetV2 VWW 80.69%77? (Channels: 64, Stride: 2)1.115 M(Channels: 64, Stride: 1)3 x 80 x 80 79,56„ / o77? (Channels: 96, Stride: 1)Conv 24 x 160 x 1 x 1FC 2 x 160Conv 6 x 1 x 5 x 5LeNet Conv 16 X 6 x 5 X 5 MNIST 98.49%Conv 120 x 16 x 5 x 50.417 M FC 84 X 120 98.83% 98.83%FC 10 X 84Conv 6 x 1 x 5 x 5MLPClassifier FC 200 X 216 F-MNIST 90.57%FC 120 x 2000.165 M FC 84 x 120 1 x 28 x 28 90.55%FC 10 x 84UCT-02225

[0118] No model compression or decomposition are used in any of the experiments because the focus is purely on computational efficiency. Graph optimizations, e.g. Layer fusion for batch normalizations, are enabled through the ONNX library for all models.

[0119] Evaluation Results. Various embodiments comprise evaluating the following research questions (RQs): RQ1 : How fast is Lupe in continuous power? RQ2: How fast is Lupe in intermittent power? RQ3: How much energy can Lupe save? RQ4: How does each of Lupe’s optimizations contribute to total performance? RQ5: How do Lupe generated programs improve LEA utilization?

[0120] How fast is Lupe in continuous power? FIG. 13 shows inference latencies of all 5 models in continuous power. In FIG. 13, the implementations shown are from left to right, for each graph: Tails, Lupe-BT, Hawaii and Lupe. MobileNetV2 does not fit in FRAM for Hawaii.

[0121] On average, Lupe achieves 12.36x, 11.1 lx, 2.22x speedup over Tails, Lupe-BT and Hawaii respectively. Also, as noted in Table 2, Lupe’s speedup comes without loss of accuracy. Lupe is significantly more efficient than the bottom-up implementations, i.e. Tails and Lupe-BT. Similar to the batched-acceleration, Hawaii also tries to utilize the SRAM as much as possible, which brings them a tremendous speed up in DS-CNN. Moreover, Hawaii partially restructures programs for cheaper logging cost in intermittent computing. Hawaii also optimizes the DMA calls in the same way as Lupe does. However, although extra performance benefits are introduced, Hawaii fails to recognize the possibility of systematically reorganizing the program to reduce LEA preparation overhead (as discussed in the section entitled Reorganize Programs to Eliminate Preparation Costs Section).Additionally, Hawaii does not implement DNN layers adaptively, contrary to the optimization in the section entitled Select Best-Performing Accelerations.UCT-02225

[0122] Overall, these results show that Lupe’s top-down approach achieves large performance improvements by enabling optimizations that are not possible otherwise, especially through bottom-up methods.

[0123] How fast is Lupe in intermittent power? FIG. 14 shows latency per inference, including device initialization time, under different reboot rates. In FIG. 14, the implementations shown are from left to right, for each graph: Tails, Lupe-BT, Hawaii and Lupe. Any solid portion at the top of any bar in FIG. 14 represents Intermittent Support Overhead. During each reboot cycle, a uniformly random reboot time may be picked in the given time intervals. As shown in the figure, the total execution time tends to be stable regardless of the reboot rate. Even in the worst case, where Tails restart 11740.64 times per inference for DS-CNN, it only adds 7.5% execution time.

[0124] For the total execution time, Lupe achieves 21.65*, 9.91 *, 2.75* average speedup over Tails, Lupe-BT and Hawaii respectively. These higher speedups comparing to Tails and Hawaii show that Lupe provides a much more lightweight intermittent-safe support.

[0125] Furthermore, the overhead of adding such support may be analyzed by highlighting the differences between continuous and intermittent safe versions. The intermittent runtime of Lupe reduces the average intermittent-safe overhead by 96.65%, 47.49%, 71.15% over Tails, Lupe-BT and Hawaii respectively.

[0126] Lupe-BT’ s bottom-up continuous implementation may be augmented with a lightweight top-down intermittent-safe support. This results an extremely cheap logging overhead. Particularly, on average, Lupe-BT reduces intermittent-safe overhead by 91.05% over Tails. Overall, these results show that Lupe maintains its speed advantages under intermittent power. In addition, the comparison of Tails to Lupe-BT highlights the efficiency of Lupe’s top-down application of loop continuation compared to a bottom -up approach.UCT-02225

[0127] For results shown in FIG. 14, all approaches may be benchmarked under different reboot rates. The overhead of adding intermittent safe support to each method may also be shown. Tails and Lupe-BT have comparable latencies under continuous environments, yet adding intermittent runtime in a top-down way for Lupe-BT leads to much lower overhead for logging information. Hawaii does not support MobileNetV2.

[0128] How much energy can Lupe save? FIG. 15 shows the energy consumption of a single DNN inference for both continuous and intermittent-safe implementations, benchmarked using TI’s energy trace software. In FIG. 15, each implementation is shown with results for both continuous and intermittent-safe operation. From left to right, for each model, the implementations are from left to right: Tails, Lupe-BT, Hawaii and Lupe. Hawaii does not support MobileNetV2. On average, Lupe reduces energy consumption by 88.06%, 83.9%, 54.13% in continuous conditions over Tails, Lupe-BT and Hawaii, respectively. Additionally, the energy consumption of intermittent- safe programs with no reboots may be measured. Lupe provides 92.3%, 79.44%, 56.92% average reduction over Tails, Lupe-BT and Hawaii, respectively.

[0129] Additionally, FIG. 16A shows power snapshots of executing ResNet3 for continuous and FIG. 16B shows power snapshots of intermittent-safe implementations: Tails 1602, Lupe-BT 1603, Hawaii 1604 and Lupe 1605. Continuous power may be used and the MSP430 may spin for 2 seconds between each inference to differentiate compute and idle time. Lupe has similar peak power as prior works, but because Lupe dramatically improves LEA utilization-as motivated in the section entitled Reducing Data Manipulation Overhead and evaluated in the subsection entitled How do Lupe generated programs improve LEA utilization? Various embodiments achieve energy efficiency gains from latency reductions in both continuous and intermittent-safe implementations. Overall, these results show that Lupe achieves not just significantly lower latency, but that it has no significant energy overhead.UCT-02225

[0130] How does each of Lupe’s optimizations contribute to total performance? The following explains how each optimization technique improves the overall performance. FIR- instantiations may be used for all convolutions unless adaptive layer generation, as explained in Adaptive DNN Layer Generation, is enabled. Each optimization is added on top of the previous ones. Overall, Lupe provides 11.11 x speedup over the non-optimized version (Lupe-BT), as shown in FIG. 17 which shows a breakdown of optimizations that contribute to the overall speedup in Lupe, starting with a FIR only instantiation and adding other optimizations step by step. DS-CNN uses a different y2-axis scale. FIG. 17 shows for each model, from left to right: No Optimization, DMA Optimization, LEA Optimization, Adaptive Data Movement, Batched Acceleration, and Adaptive Generation.

[0131] The discussion in the sections entitled Reorganize Programs to Eliminate Preparation Costs and Generate Efficient Data Movement Section demonstrate how to minimize DMA and LEA preparation costs by reorganizing programs. On average, optimizing DMA and LEA function calls improves latency by 1.24x and 2.06x, respectively. Generating data movement methods adaptively only adds 1.04x speedup because DMA operations are already efficient from previous optimizations. Lupe uses batched LEA operations to reduce invocation overhead.

[0132] Batch Acceleration to Reduce Invocation Costs. Changing FIR-instantiations to batched-FIR-instantiations brings 2.12x speedup. Finally, having the possibility of using other instantiations, namely adding the Calibration phase, gives a speedup of 1.74x. This speedup mainly comes from small input sizes of last few convolution layers.

[0133] Various embodiments may comprise achieving a massive 27.85x speedup on DS- CNN, that comes from batched acceleration, improving performance by 4.4x. DS- CNN has large input and output channels for every layer, which provides a perfect execution scenario for batched instantiations.UCT-02225

[0134] Overall, these results show that each of Lupe’s optimizations are meaningful to the overall result. Even the adaptive data movement, that has the smallest impact, provides 4% performance improvement on average.

[0135] How do Lupe generated programs improve LEA utilization? In several places, this disclosure asserts that the top-down approach will improve LEA utilization, and therefore overall performance. Various embodiments may measure the LEA utilization by comparing LEA execution time of all LEA functions, i.e. time of msp lea invokeCommand in FIG. 10, with the total inference time for one input. In other words, various embodiments may compute LEA utilization as the percentage of time that the LEA is computing during execution; i.e. the ratio of computation time using the LEA to total time for inferences. Various embodiments may compare the utilization of two instantiations, FIR-instantiations and MAC -instantiations, throughout every DNN layer precluding the Calibration phase. As shown in FIG. 18, Lupe greatly improves LEA utilization: on average from 12.70% to 41.44% for FIR-instantiations and from 7.22% to 32.88% for MAC-instantiations. In FIG. 18, for each DNN shown on the horizontal axis, the left-side bar shows LEA utilization for FIR- Inst. (bottom up) on the lower portion of the bar and FIR-Inst (top down) on the upper portion of the bar. The right-side bar for each DNN shows MAC -Inst, (bottom up) on the lower portion of the bar and MAC -Inst (top down) on the upper portion of the bar.

[0136] Overall, these results show that the initial hypothesis is correct: the top-down approach leads to better LEA utilization and better overall performance.

[0137] Top-down optimizations: A rich set of prior works have explored top-down optimizations in high power devices, i.e. GPUs. However, none has explicitly explored the challenges of the top-down optimization on ULP devices, with one exception. Hawaii employs top-down checkpoint insertion for intermittent-safe DNNs on ULP. Lupe is the first work that attempts to tackle the unique challenges of applying a top-down approach to overallUCT-02225DNN optimizations on ULPs. Specifically, programming models and optimization goals differ significantly from a GPU to the LEA. The GPU optimization requires minimizing costs of running parallel computation across different execution units. The LEA, on the other hand, has high costs for SRAM preparation and function invocation within a single execution unit and hence requires novel solutions to reduce these costs and increase its utilization.

[0138] DNN inferences on ULP devices: To support such efficient inference, several prior works have proposed schemes to deploy DNN models on ULP devices, for MSP430 and other hardware. Some use CPU-based implementations, while others accelerate DNN programs through on-chip accelerators. However, they all fail to systematically structure programs that utilize on-chip accelerators from a top-down view and thus suffer from the limitations analyzed in this disclosure. Particularly, various embodiments optimize the model architecture itself through quantization and neural architecture search. Lupe, in contrast, does not change the DNN architecture and optimizes it for the LEA by minimizing data movement and adaptively generating DNN layers.

[0139] Intermittent DNN systems: Many works support efficient, timely DNN applications in intermittent power. Some focus on DNN architectures, while others use advanced scheduling algorithms. Nevertheless, these systems require the underlying DNN inference systems to be efficient.

[0140] Other embodiments. Lupe’s top-down approach can be applied to other microcontrollers with similar accelerators because it only uses common linear algebra operations. Moreover, other DNN operations, such as self-attention, can be added to Lupe using the same top-down methodology. For example, self-attention can be accelerated by matrix-matrix multiplication or MAC functions on the LEA.

[0141] This disclosure describes Lupe, a code generation framework for DNN inference on ULP devices. Lupe uses a top-down approach to achieve three specific advantages. First,UCT-02225Lupe greatly reduces the data manipulation cost for accelerator use. Second, it adaptively generates the best performing instantiations of DNN layers. Finally, Lupe implements an extremely low cost intermittent-safe computing runtime. For two prior works, Lupe achieves an average speedup of 12.36* and 2.22* in continuous power, reducing the intermittent overhead by 96.65% and 71.15%, respectively.

[0142] SLUMBER

[0143] In another aspect, disclosed herein are methods and systems employing a deep neural network (DNN) model 1900 that can change width during inference (e.g., using one width for one layer, and using a different width for a later layer) as shown in FIG. 19. The DNN 1900 includes a plurality of processing elements shown as PEI-1, 1-2, . . ., N-M which are organized in a plurality of rows (where a single row is for example PEI-1, 2-1, 3-1, 4-1) and columns (where a single column is for example PE2-1, 2-2, 2-3, 2-4). The processing elements in one stage (column) may provide an output to one or more processing elements in the next stage (column). The changeable width and depth can be combined with the ability to exit early, shown at 1902; i.e., produce an inference output 1904 after executing only early layers and ignoring later layers. This combination allows inference depth and width to be dynamically changed while inference is executed. For example, depth may be adjusted by removing a column 1906 from the DNN during inferencing. Width may be adjusted by removing one or more processing elements such as shown at 1908. The choice of depth and width may be externally controllable 1910 so that an external entity (rather than the inference itself 1912) can determine the depth and width during inference. Also disclosed is a method for training such a network (that can alter width and depth during inference) such that its accuracy at any width / depth combination is equivalent to a standard network (that cannot change) of the same width and depth. Specifically, the rate at which widths are shrunk may be grouped to have separate batch-norm stats. For instance, for drop rates from 20-100%, 9UCT-02225 different batch-norm statistics may be involved. When dropping width entire channels may be dropped. Rows or columns may also be dropped albeit with lower efficiency. Accuracy is improved with more efficient flop reduction. When training with dropout, each input may be used multiple times in the same batch. The best and worst models may be trained without any dropout. Also disclosed is a method by which the aforementioned network may be designed to operate on currently available hardware, which may not be able to execute all layers at maximum width in a timely manner. But such a network can be run on future hardware that does produce a timely result with no changes and provides more accurate results (as opposed to retraining a network for the new hardware).

[0144] Also disclosed is a computerized method for tracking execution of a long-running task using energy consumption in a noisy environment. This is applicable to any long-running task (not just inference) which has predictable energy consumption but runs in a noisy environment where energy consumption cannot be directly measured. The foregoing approach can be extended to other methods that are subsets (e.g., random forest) or super sets (e.g., transformers) of typical neural network models.

[0145] In accordance with the foregoing, disclosed herein is a computer-implemented method for performing inference using a neural network model. The method includes modifying width of the neural network during inference whereby a first width is employed for a first layer and a second width is employed for a second layer. The method may also include performing an early exit of the neural network whereby one or more layers of the neural network are not employed in generating an output of the inference. The method may further include accepting one or more inputs provided by an entity external to the neural network to select depth and width of the neural network in performing inference.

[0146] Also disclosed herein is a computer-implemented method for training a neural network model characterized by changeable width and depth during inference where theUCT-02225 method includes shrinking width at a rate wherein a grouping of the rate is characterized by separate batch-norm statistics and dropping an entire channel when shrinking width.

[0147] Overview of Slumber. Batteryless embedded systems must perform accurate, deadline- driven neural network inference despite unpredictable power interruptions from environmental energy harvesting. Prior work has explored energy-adaptive inference that relies on static schedules or fixed model variants to balance accuracy and energy use. However, these approaches either (1) require brittle profiling tuned to specific conditions (e.g., assuming energy availability) or (2) cannot adapt if energy availability changes midinference. A key insight described herein is that robust, deadline-driven inference requires both (1) a richer set of accuracy / latency tradeoffs in the neural model and (2) dynamic adaptation to fluctuating energy conditions, conditions. Various embodiments implement these insights as SLUMBER, a profiling-free system that combines a DNN architecture, called EarlySlims, with a Progressive Adaptive Control Engine (PACE), a scheduler that dynamically selects and adjusts inference accuracy / latency tradeoffs mid-inference. SLUMBER is the first system known to support mid-inference adaptivity on real hardware, and compared to prior adaptive inference it improves accuracy by up to 15 percentage points, reduces deadline violations by 10x and requires no environment or hardware-specific tuning to deploy. These results highlight SLUMBER as a robust, deployable solution for real-time, deadline-aware inference on battery-free embedded sensors.

[0148] Intermittent systems powered by energy harvesting offer an appealing platform for long-term, maintenance-free sensing and neural network inference at the extreme, low-power edge. However, these devices must operate within unpredictable environmental energy conditions, which make it difficult to execute long-running tasks (such as neural network inference) that require tight control over runtime latency while demanding high accuracy.UCT-02225Applications such as object recognition, keyword detection, and event alerts often have fixed latency constraints and missing these deadlines leads to degraded quality or mission failure.

[0149] Ensuring accurate, deadline-aware inference under intermittent power and limited on- device memory remains a core challenge in this area.

[0150] Prior work has addressed the need for adaptive neural network inference techniques that trade accuracy. Due to device memory constraints, these tradeoffs need to be present within a single model and two methods have emerged for constructing a single network with accuracy / latency tradeoffs. The first, Early Exits (also known as BranchyNet, MultiExit, or Anytime networks ). are a set of techniques for augmenting networks with multiple, predefined exit points. These allow inference to stop at progressive network depths, so when energy is scarce, the network can simply use the most recent exit point. However, they can only provide a limited range of tradeoffs due to architectural constraints. The second set of techniques, Slimmable Networks, allow the network to be modified in width. These techniques provide a wider range of tradeoffs but require the path through the network to be set when inference begins and do not allow it to change mid-inference. Early Exits and Slimmables have been combined into SLEXNet, which provides a much richer tradeoff space than Early Exit models, but retains a key disadvantage of Slimmables: the path through the network must be selected when inference starts and cannot change during execution. As a result, prior approaches are brittle: they rely on either known, stable energy during inference or design-time profiling to select the right path through the network, operating as designed so long as these assumptions are not violated during deployment. But, since energy conditions are unpredictable and inference is long, energy availability can and does change during inference. Furthermore, approaches based on design-time profiling are time-consuming and only perform as expected when the design and deployment conditions match. This makesUCT-02225 existing approaches hard to deploy in the field where energy conditions vary because training and profiling for every hardware setup and energy condition is either costly or infeasible.

[0151] A key insight disclosed herein is that robust and deployable inference in intermittent systems requires both (1) a rich set of accuracy / energy tradeoffs and (2) an adaptive strategy to dynamically adjust to fluctuating energy availability — mid-inference — without relying on design-time profiling of energy conditions. To achieve this, inference and runtime must support both fine-grained tradeoffs within a single model and continuously monitor energy conditions to react during inference while quickly adapting to deployment conditions on device.

[0152] Various embodiments may implement this insight in SLUMBER, a complete system for deadline-aware inference on intermittently powered sensor systems that does not rely on design-time profiling or assumptions of energy availability. SLUMBER combines a neural network architecture that exposes a rich trade-off space with the Progressive Adaptive Control Engine (PACE), a low-overhead runtime scheduler that dynamically estimates inference latency. Various embodiments may call the neural network architecture EarlySlims because it merges the structural flexibility (z.e., the ability to change execution path during inference) of Early Exit models with the fine-grained tradeoffs of Slimmable networks. PACE uses its energy-aware mid-inference strategy to change the network’s execution path dynamically, ensuring deadlines are met even when the energy availability drops unexpectedly. SLUMBER is thus the first system (to the knowledge of the inventors) that supports an adaptive mid-inference strategy for intermittent systems without requiring design-time profiling. This capability is enabled by the combination of EarlySlims (the mechanism enabling mid-inference adaptation) and PACE (the policy governing adaptation during inference).UCT-02225

[0153] Various embodiments comprise implementing SLUMBER on a TI MSP430 microcontroller and evaluate it across three MLPerf Tiny workloads and 20 real -world volatile and non-volatile energy traces. Various embodiments comprise comparing to prior approaches (including both Early Exits and Slimmables). Overall, various embodiments comprise finding that SLUMBER improves inference accuracy by up to 15 percentage points, reduces deadline violations by 10 X, requires no static profiling or tuning, and has low memory overhead, making it practical for deployment across diverse embedded environments.

[0154] Various embodiments comprise designing EarlySlims, a DNN architecture with rich energy-accuracy tradeoffs suitable for resource-constrained, energy-harvesting devices and support for mid-inference adaptation. Various embodiments comprise developing PACE, a profiling-free, energy-aware inference scheduler using efficient latency predictions based on real-time recharge observations. PACE implements a novel Energy-Aware, Progressive strategy, which allows it to change the EarlySlim inference paths mid-execution, preserving deadline guarantees under energy volatility. Various embodiments comprise evaluating SLUMBER on real hardware and show that it improves accuracy and timeliness without manual tuning, establishing a path toward robust, deployable intermittent inference systems.

[0155] To deploy robust inference systems on intermittent embedded devices, various embodiments must address challenges across three dimensions: (a) how to construct neural network models that offer a rich range of accuracy / latency / energy tradeoffs, (b) how to execute these correctly (including with respect to deadlines) under frequent power failures, and (c) how to schedule inferences to meet deadlines without relying on design-time profiling and assumptions, which may not remain valid in deployment. This section surveys prior work in each of these three areas, highlighting why existing solutions fall short in energy-volatileUCT-02225 deployments. The key principles behind SLUMBER are then motivated, which addresses these challenges with an adaptive model and runtime scheduler.

[0156] Embedded, Adaptive DNN Inference. To meet neural network inference deadlines under intermittent power, DNN models must offer configurable tradeoffs between accuracy, latency, and energy. Two distinctive adaptive architectures have emerged to support these tradeoffs: Early Exits and Slimmables.

[0157] Early Exits modify network depth to introduce multiple, intermediary exit points and thus enable inference to terminate early if necessary. Earlier exits are cheaper (in time and energy), while deeper exits are more accurate. Because execution proceeds progressively (from one exit point to the next), these models can opportunistically provide the best result before a deadline, even when energy is volatile. However, this progressive approach is limited: creating exits requires exponential scaling in computing between exits, which is impractical on memory and compute -constrained embedded devices. Hence, Early Exit networks for embedded systems typically offer coarse-grained trade-off spaces, which limit their ability to provide high-accuracy inference within a deadline (see, Q2: Tradeoff Space Richness).

[0158] Slimmables, in contrast, modify network width by enabling only a subset of channels per layer. Given the wide width of modern neural networks, this approach offers a finer set of accuracy / latency tradeoffs (see, Q2: Tradeoff Space Richness). However, Slimmables require the configuration (z.e., the switch) to be fixed at inference start and cannot adapt midexecution. While not an issue in continuously powered systems, this is a problem for intermittent systems as the available energy may worsen mid-inference, which would cause the inference to fail to complete within its deadline.

[0159] SLEXNet combines both Early Exits and Slimmables to improve the tradeoff richness without requiring extremely deep networks. However, SLEXNet runs on batched images andUCT-02225 is designed for continuous video processing using a relatively high-power system (NVIDIA Jetson Orin), going beyond the memory and energy constraints of current intermittent systems. It also requires that the network path be selected at the start of inference and does not support changing the path during execution. Given these assumptions, SLEXNet’s runtime and architecture design do not support inference on energy-harvesting intermittent sensors.

[0160] Scheduling Under Intermittent Power and Deadlines. Batteryless systems collect energy from ambient sources and store it in small energy buffers such as capacitors. Since harvested energy is rarely sufficient for continuous operation of intermittent tasks, batteryless systems’ operation is characterized by periods of activity interspersed with recharging periods, posing several challenges for ensuring correct execution. Correctness is further complicated when a task must not just produce the right answer but do it by a given deadline. The core contradiction is that intermittence implies unpredictable interruptions, while deadline-based execution assumes predictable resources. Several challenges follow: (i) Energy unpredictability: Harvested energy varies with environmental conditions (e.g., light, RF availability), making it difficult to determine how much energy will be available in the future, (ii) Long-running tasks: Many useful tasks (including the subject of this disclosure: DNN inference) span multiple power cycles. Ensuring their correctness requires careful checkpointing and progress tracking, (iii) Data Freshness and Deadline-awareness: Some sensing applications-e.g., object recognition, keyword detection, and event alerts require not just eventual correctness, but bounded latency to ensure the data returned is useful to the consumer.

[0161] Prior Scheduling of Intermittent Inference with Deadlines. Recent work has attempted to bring deadline-aware scheduling to intermittent DNN inference by combining adaptive model architectures with runtime energy monitoring. However, these solutionsUCT-02225 typically rely on design-time profiling, assume stable energy conditions, or support only limited adaptivity.

[0162] Catnap and Culpeo offer intermittent, scheduling primitives (e.g. recharge detection, task scheduling). However, they only support deadlines for short tasks that complete within a single discharge and assume that long-running tasks do not have deadlines. Celibi improves scheduling by adjusting the system’s active and recharge durations based on runtime energy availability. However, it cannot adjust its task to changing energy conditions. Furthermore, it assumes that no energy is harvested during executions, which is reasonable for short tasks but leads to inconsistencies (and either a loss of accuracy or missed deadlines) for long-running tasks, such as DNN inference.

[0163] REHASH adapts its execution to improve throughput, sensor coverage, and completion rates. While REHASH enables a variety of flexible adaptation strategies without using any hardware measurements, the core of the adaptation relies on a heuristic function developed through design-time profiling, which may not hold in real-world deployment scenarios.

[0164] While REHASH explores heuristics for degrading DNN networks, ePerceptive, and Zygarde propose managing the accuracy / energy tradeoffs by altering the DNN architecture via Early Exits. They schedule inference by pre-selecting DNN exits based on energy profiles collected during design-time. This makes them brittle: if conditions during deployment differ from design-time (due to changes in capacitance, temperature, ambient energy, or deadline) then the scheduler will either fail to meet deadlines or under-utilize the available energy. Specifically: (i) ePerceptive uses a lookup table indexed by recharge duration to choose an exit, but this table must be re-generated if the model, hardware, or environment changes, (ii) Zygarde partitions the DNN into mandatory and optional segments. While mandatory segments are always executed, optional ones are scheduled if a design-time learned valueUCT-02225 indicates energy availability before the deadline. This learned factor is derived from longterm profiling and must be recomputed for new deployments, limiting generalizability.

[0165] The latter systems (REHASH, ePerceptive and Zygarde) also lack an Energy-Aware, Mid-inference Strategy — the ability to adjust execution paths dynamically as energy fluctuates. ePerceptive utilizes only a fixed execution path and misses deadlines when energy conditions deteriorate during inference. Zygarde, on the other hand, progressively and greedily executes every single exit. With this progressive strategy, Zygarde can withstand deteriorating energy conditions at the cost of unnecessary calculations under stable energy conditions (see, Q4: Mid-Inference Strategy).

[0166] Motivation for Slumber. The limitations of prior work reveal a clear gap: supporting deadline-aware, intermittent inference requires both adaptivity and robustness to variability — in hardware, energy, and workload conditions. Three critical properties such a system must satisfy may be identified. (1) A Rich and Flexible Tradeoff Space: The system must support a wide range of energy / I atency / accuracy tradeoffs within a single model to ensure efficient operation across changing energy conditions or even deadline constraints. (2) Deployment- Agnostic Scheduling: The system should avoid reliance on design-time profiling, hardwarespecific tuning or unchanging, learned energy models. It should generalize across conditions with minimal reconfiguration. (3) Energy-Aware, Mid-Inference Adaptivity: Rather than selecting a fixed configuration when inference starts, the system must adjust its behavior during execution based on real-time energy observations, enabling graceful, efficient degradation under volatile energy conditions.

[0167] Table 3 summarizes the differences between SLUMBER and prior work:System Power Tradeoff Design-Time Mid-inferenceModel Richness Profiling Strategy / ScheduleCatnap Intermittent N / A No N / ACulpeo Intermittent N / A Hardware N / AUCT-02225Celibi Intermittent Fixed Per-task Hardware & N / A SoftwareREHASH Intermittent Coarse Software N / A ePerceptive Intermittent Coarse (Early Energy & Fixed Exits ) InferenceZygarde Intermittent Coarse (Early Energy (Optional) Greedy ProgressiveExits)SLEXNet Continuous Rich (Early + Hardware & FixedSlimmable) SoftwareSLUMBER Intermittent Rich No Energy-Aware,(EarlySlim) Progressive

[0168] Table 3 shows that prior systems fall short along one or more of these axes. Early Exits support adaptive mid-inference strategies but lack rich tradeoffs. Slimmables offer fine tradeoffs but cannot adapt mid-inference. Systems like ePerceptive and Zygarde use designtime profiling to adapt, so they cannot handle deployment conditions that were not accounted for at design time. These limitations lead to either deadline misses or poor energy utilization (z.e., lower accuracy answers due to energy underutilization).

[0169] SLUMBER addresses these challenges with a new combination of model architecture and runtime scheduling. It introduces EarlySlims, which provide both fine-grained tradeoffs and support mid-inference adaptation, with PACE, an online, runtime scheduler that makes fine-grained, per-layer decisions about when to exit the network based on available energy and deadlines. SLUMBER assumes neither fixed conditions nor design-time tuning / profiling, making it uniquely suited for real-world intermittent deployments. Various embodiments may introduce SLUMBER (FIG. 20), an adaptive intermittent inference framework that provides deadline-aware, high accuracy inference despite unknown, volatile energy conditions. SLUMBER achieves its goal using the EarlySlim architecture (Fig. 21) and the novel Progressive Adaptive Control Engine (PACE) runtime (Fig. 22). FIG. 21 shows EarlySlims networks which can adapt both their width (along the columns) and depth (along the rows), providing a rich tradeoff space and enabling mid-inference adaptation. Wider and deeperUCT-02225 networks are more accurate but require more computational / energy resources. Active and inactive network paths are shown respectively in black and gray. FIG. 22 shows an overview of PACE during inference; PACE continuously queries the estimation algorithm during inference to ensure inference completes without a deadline violation.

[0170] SLUMBER utilizes the EarlySlim adaptive architecture, which - unlike Early Exits - provides a rich tradeoff by altering the width and depth of network, and - unlike Slimmables - supports mid-inference adaptation along the network’s depth.

[0171] PACE employs latency estimation algorithm to predict the la- tency of inference tasks. The underlying hardware of the intermit- tent system harvests ambient, volatile energy and stores it in the system capacitor. PACE monitors the recharge time of this capacitor as a proxy for the current rate of incoming energy. This estimation algorithm predicates its estimates on the relationship between this deployment-time incoming energy and inference progression. This relationship is learned online and adapts to varying deployment conditions, all without any design-time profiling.

[0172] PACE’s controller adapts inference to energy conditions by leveraging Early Slims and the latency estimation algorithm. When an inference job arrives, the controller estimates the latency of each tradeoff point based on recent energy conditions and executes the most- accurate network path that would complete before the dead- line. Moreover, PACE adapts to mid-inference energy fluctuations by monitoring the progression of inference and the current rate of incoming energy. If the controller estimates that the current network path would lead to a deadline violation, PACE alters the execution path to an earlier exit, ensuring timely inference completion.

[0173] Overall, SLUMBER maximizes inference accuracy under a strict deadline with unknown and changing energy. EarlySlim, PACE’s estimation algorithm, and PACE’s control engine networks were analyzed.UCT-02225

[0174] Hardware & Software Support. Various embodiments comprise implementing SLUMBER on a TI MSP430FR5994 Microcontroller Unit (MCU), a popular chip in the intermittent community. This chip has 256KB of FRAM, 8KB SRAM, up to 16 MHz of CPU clock speed, and a low energy accelerator (LEA), but importantly provides no hardware support for floating point numbers or integer division.

[0175] SLUMBER is built using existing open-source packages for Deep Neural Network (DNN) libraries and checkpointing across power failures. Various embodiments may extend the checkpointing system to add DNN support and make those changes available with the rest of the code. Similar to prior work, the MCU’s ADC may be utilized to measure voltage, its internal comparator to detect voltages, and its clocks to keep track of time. When recharging, the device is placed in low power mode to minimize energy consumption. The following describes how EarlySlims may be efficiently implemented and the novel properties of the PACE, which include the on-device latency estimation and mid-inference adaptivity.

[0176] EarlySlims. Exemplary embodiments are EarlySlim networks (FIG. 21), which can alter both depth and width of their execution path, merging the structural flexibility of Early Exit models with the rich tradeoffs of Slimmable networks, all within a single model. Alternatively, a rich tradeoff space can be achieved using an ensemble of networks. However, a fine-grained ensemble of models is practically impossible due memory constraints of energy harvesting devices (e.g. 256 KB on the MSP430). Exemplary EarlySlim architecture, model training, and MSP430 implementation is described herein.

[0177] Motivation & Design. The EarlySlims architecture is motivated by two key insights. First, a rich and flexible energy-accuracy tradeoff space - i.e. one with a greater, wider spanning Pareto-optimal operating points - improves energy utilization and accuracy of an inference across a range of energy conditions. Second, the ability to alter the execution path mid-inference allows the inference system to adapt to unforeseen deteriorating energyUCT-02225 conditions. These properties may be achieved by combining the Early Exit and Slimmable architectures (FIG. 21).

[0178] By altering both depth and width, the EarlySlim architecture exposes a richer trade off space. Thinner, shorter paths enable less accurate, yet extremely cheap inference results. In contrast, wider and deeper paths provide the highest possible accuracy when energy is abundant. This rich tradeoff space has two benefits. To begin with, EarlySlims produces a multiplicative increase the number of potential tradeoff points, increasing the fine-granularity of system and how well the system matches energy conditions to tradeoff points. Second, EarlySlims provide extremely cheap tradeoff points, enabling operation at scarce energy conditions. The richness of EarlySlim’ s tradeoff space may be compared along these properties to both Early Exits and Slimmables in the subsection entitled, Q2: Tradeoff Space Richness.

[0179] EarlySlims also retain an important benefit of Early Exit networks: progressive execution of exit points. Hence, the network can always shorten or increase the depth of the network mid-inference to adapt to volatile energy conditions. SLUMBER utilizes this property in its Energy Aware Progressive Exit Strategy.

[0180] Model Training. EarlySlims may be trained by combining methods from both Early Exit and Slimmable networks. The section entitled Inference Applications provides more details on the general training process.

[0181] To train along arbitrary network widths and boost the final accuracy, the techniques from the Universally Slimmable Networks (US-Nets) may be utilized: switch-privatized BatchNorm statistics, the sandwich rule, and in place distillation. In Slimmable networks, different networks switches have different BatchNorm statistics. To prevent shared statistics from reducing inference accuracy, US-Nets use switch-independent statistics. Next, instead of training every potential width, US-Nets train the model at smallest, largest, a number ofUCT-02225 other random widths (2 other may be chosen) every epoch. This technique, dubbed the sandwich rule, implicitly optimizes all subnetworks. Lastly, inplace distillation transfers knowledge from the large, full-width subnetwork to all other subnetworks. In other words, the Lslim may be defined, i. e. the loss of the Slimmable network aswhere L is the cross-entropy loss function, y is the ground truth, yiargest is the predicted labels at the largest width, and ysis the predicted labels of all other switches.

[0182] For Early Exit paths, various embodiments may find that simply applying pooling and fully connected layers to intermediate neurons resulted in poor Early Exit accuracy. Instead, each Early Exit block repeats the network’s basic block followed by a fully connected layer (e.g. Early Exit blocks in MobileNetVl networks consist of a depthwise separable convolution followed by fully connected layer).

[0183] The network may be optimized using a weighted loss function LmodeiN Lmodel= V WnLslim nn=l where N is the number of exits, wnis the weight of each exit, Lsiim, n is the Slimmable loss (Eq. 1) ending at exit n. Lmodei combines the loss of every possible subnetwork, optimizing all at once. The weights indicate the importance of each exit, balancing exit accuracies during the optimization pass. A neural architectural search may be conducted to find the most optimal Early Exit locations and loss weights, optimizing for accuracy and tradeoff space richness.

[0184] Implementation. Various embodiments may leverage Lupe’s DNN libraries and add slight modifications to support the JIT checkpointing described in Catnap. Lupe utilizes the on-chip LEA and DMA accelerators for efficient execution of convolutional and fullyUCT-02225 connected layers. Additionally, the convolutional and fully connected kernels may be slightly modified to support efficient operation of Slimmable switches. No kernel changes were required for Early Exit support, which were implemented similar to ePerceptive.

[0185] During inference, intermediate EarlySlims neurons in each feature map must be scaled and biased separately by the switch-privatized BatchNorm statistics, i.e. the neurons need to be scaled differently based on the width of the network. While such scaling and biasing is inexpensive on modern GPUs, the scaling may be found to be prohibitively expensive on the MSP430 chip. This is due to lack of hardware accelerated left shift on the MSP430 and the relatively large size of the intermediate feature maps.

[0186] To reduce this overhead burden, Lupe’s DNN kernels may be optimized in two steps. First, instead of scaling entire feature maps, the kernel weights may be scaled. Depending on the network, this optimization reduces the number of required operations by 2 — 10 X. Next, the scaling factors for each layer may be normalized to be in the [-1, 1) range and the layer weights adjusted to maintain correct operation. This normalization enables hardware accelerated scaling of layer weights without the need for any left shifting. With these optimizations, almost negligible overhead for executing Slimmable networks on the target MCU may be observed, as shown in FIG. 27 which shows accuracy and latency of Pareto- frontier of Early Exit, Slimmable, and Early Slim networks when running on constant power. Note that inference latency has a direct relationship with inference energy consumption. EarlySlim networks provide a fine-grained tradeoff space and a lower minimum energy threshold.

[0187] PACE: Progressive Adaptive Control Engine. PACE is an online, runtime scheduler that alters the depth and width of the current inference to adapt to energy conditions, maximizing accuracy and reducing deadline violations. PACE is composed of two components: (a) a latency estimation algorithm that predicts inference task latency basedUCT-02225 on recent, real-time energy conditions and (b) a controller that schedules EarlySlims and adapts the network exit to mid-inference energy fluctuations. The implementation of PACE accomplishes these objectives as efficiently as possible despite the limitations of the MSP430 architecture.

[0188] Table 4Notation DefinitionTi Required wall-power uptime for inference (ms) trLatest recharge time (ms) ttHow long inference runs between recharges (ms)E () Learned factor based on trR Number of needed recharges (unitless)D Inference deadline (ms)P[] Tradeoff points listed in decreasing accuracyBias Bias value in the control engine (ms)Subnetwork Selected Tradeoff point / subnetwork

[0189] PACE does not rely on any design-time profiling to operate. PACE learns during deployment and utilizes the learned information to adapt to varying energy conditions. While the estimation algorithm assumes that future energy conditions will resemble recent ones, the controller operates without this assumption, affording PACE the ability to adapt to midinference energy fluctuations (Q4: Mid-Inference Strategy).

[0190] Aspects of PACE and their implementation on the MSP430 are discussed. FIG. 22 depicts an overview, the algorithm shown in FIG. 24 outlines the PACE pseudocode, and Table 4 lists the used notation. Methods and systems employing the operations shown in FIG. 24 are able to change the path taken through the neural network during execution.

[0191] Latency Estimation Algorithm.

[0192] FIG. 33 illustrates algorithms in accordance with one or more embodiments of the present disclosure. Lines 1-5 of FIG. 33 is a latency estimation algorithm that determines the number of required recharges and expected latency of an inference task based on recentUCT-02225 deployment-time energy conditions accurately and efficiently, all without any design-time profiling of energy conditions. To reduce computational complexity and the need for energy harvesting modeling, the algorithm assumes that near-future energy conditions will resemble recent ones (PACE’s controller avoids this assumption as described previously).

[0193] Insights. The estimator algorithm relies on two main insights. First, in a JIT checkpointing system such as Catnap, the total time spent on the same task Ti is constant, regardless of energy conditions. FIG. 23 depicts latency of an intermittent inference task - divided into Ti and recharging time - across a range of energy conditions (Low energy conditions require more recharges). As the number of needed recharges increases, the latency of the inference task increases as well. However, the increase in latency is due only to the increase in recharging time, not Ti. In FIG. 23, Latency of an inference task in a JIT checkpointing system across a range of energy conditions. While time spent recharging increases as the number of recharges increase, the time spent on inference (T i ) remains consistent. Checkpointing time was negligible and is not shown in the figure.

[0194] Second, the recharging time - i.e. the time spent recharging the capacitor (tr) - is a reasonable proxy for current energy conditions and has been used by Catnap, Culpeo, and REHASH to estimate the incoming energy. ePerceptive also uses the recharging time to schedule its inference tasks.

[0195] Estimation. As depicted at line 4 of FIG. 33, Given these insights, latency of an inference task can be described as the latency of the task under wall power (Ti ) plus the number of required recharges (R) multiplied by the average recharging time (tr). The number of required recharges can be estimated as follows. Let ti be how long the inference task runs between two recharges under current energy conditions. As depicted at line 3 of FIG. 33, a task requiring Ti uptime requires individual recharges. Since ti is unknown beforeinference begins (when a trade-off point must be selected), PACE uses an ExponentiallyUCT-02225Weighted Moving Average (EWMA) algorithm to learn ti based on the current recharging time tr, which is a reasonable proxy for current energy conditions. Details are provided in the section titled Implementation.

[0196] Control Engine. PACE’s control engine adapts to real-time energy conditions by utilizing an energy-aware, iterative strategy. As illustrated at lines 8-11 of FIG. 33, when an inference task arrives, the control engine first queries the estimation algorithm to find the latency of each potential tradeoff point and schedules the most accurate network path that would complete before the deadline under current energy conditions. When selecting this initial network path, the engine can adapt the EarlySlim network in both width and depth.

[0197] As shown at lines 1-5 of FIG. 33, while the estimation algorithm assumes near future energy conditions will resemble recent ones, this assumption may not hold in practice. In scenarios where future energy is less than assumed, executing the initially selected network path could result in a deadline violation.

[0198] As shown at lines 12-18 of FIG. 33, PACE avoids such scenarios by continually monitoring energy conditions during inference. After the execution of each DNN layer, PACE queries the estimation algorithm again. If the new latency estimate - which is based on energy conditions during inference - indicates a deadline violation, PACE adjusts the network to a less expensive exit. With this design, PACE avoids unnecessary execution of intermediate exit paths unless required by mid-inference energy fluctuations, unlike the greedy progressive strategy which executes all exit paths regardless of energy conditions.

[0199] Implementation. PACE’s MSP430 implementation focuses on two aspects: (1) efficient and accurate estimation of required recharges ( / ?) and Latency (L) and (2) efficient and accurate operation of the PACE control engine.UCT-02225

[0200] Recharge Estimation. To estimate R (Eq. 3), PACE uses an Exponentially MovingAverage Algorithm (EWMA) algorithm to learn E (), the relationship between trand ti at deployment-time (Eq. 4).

[0201] The implementation groups potential trvalues into 128 different groups and uses a lookup table to access the values corresponding to each tr, updating them based on Eq. 4 each recharge cycle. The size of the lookup table can be adjusted based on capacitor size to tradeoff memory overhead and accuracy.

[0202] EWMA Rationale. PACE utilizes the EWMA algorithm for variety of reasons. First, EWMA is a reasonable predictor of near future (5-10 seconds) energy because it is low-pass HR filter with exponentially decreasing weights assigned to older samples. Using information from the recent past, the latency algorithm can predict near future behavior while filtering out high-frequency, systematic noise and adapting to stable trends. While the latency estimation can be combined with longer-term energy prediction, it was found that near future energy prediction to be sufficient (QI : SLUMBER’S Improvement to Accuracy and Timeliness).

[0203] Equally important is the fact that Eq. 4 can be efficiently implemented on the MSP430. The MSP430 does not have any hardware support for floating point numbers or integer division and the learning algorithm must be updated every recharge cycle to account for changes in the environment. Thus, any potential algorithm must be efficient with minimal hardware support. The EWMA algorithm can be implemented efficiently using fixed point arithmetic and integer shifting as long as a = 2 — N .

[0204] Finally, meticulous selection of a is vital for ensuring the EWMA algorithm filters out systematic noise while adapting to trends. As discussed, important step in PACE’sUCT-02225 latency prediction is estimating R. Given the ceiling operation in Eq. 3, under- and overcounting the number of recharges can drastically impact the estimated latency (L). Underestimating the number of required recharges results in overpredicting available energy and potentially missing the inference deadline. Overestimating, on the other hand, would allow for more conservative estimates and potentially less accurate tradeoff points. However, the observed trand ti values used in calculating EQ) (and by extension R) are inherently noisy (from systematic and hardware sources), which can impact the accuracy of the recharge estimation. A reasonable a would filter out this high-frequency, irrelevant noise while factoring in relevant trends to minimize the under- and overestimation of needed recharges.

[0205] To find such an a value, an experiment may be conducted with 5 different a values. One of the inference applications (CIFAR-10) may be run under 10 non-volatile energy conditions (both described in below in Experimental Methodology). Each a value test is run with an unpopulated lookup table and was given 10 inference tasks per energy condition to warm-up (i.e. all a were given the same amount of time to converge). Afterward, 10 inference tasks per energy condition may be run and the recharge count underestimation 2501 and overestimation 2502 in percentages shown in FIG. 25. As illustrated, larger values overestimate less often but underestimate quite frequently. In contrast, smaller a values rarely underestimate but are prone to overestimate the number of needed recharges. Given these results, 25may be selected as a reasonable a that filters out the systematic noise and minimizes imperfect estimates of R.

[0206] Warm-up. PACE’s operation as outlined in Eqs. 3 and 4 requires two distinct warmup phases. First, PACE needs to populate its lookup table, which is filled with l’s at first boot. However, given that choice of a and the experiment conducted in the previous paragraph, the lookup values for any energy condition should converge and operate optimally within 10 inference tasks. Further, the Ti values are unknown for all tradeoff points. While TtUCT-02225 values are constant and can be measured at design-time, an extra step may be taken to measure them online to avoid design-time parameters. These values may be measured online at first boot, which can take anywhere between 1 and 10 minutes, pending initial energy conditions.

[0207] Control Engine. A DNN engine that executes DNN layers may be implemented based on the tradeoff point definition. This engine manages the limited device memory and executes a variety of layers with varying widths and depths to support the Early Slim architecture. The EarlySlim support adds slight runtime overhead when compared to prior work. The majority of this overhead is due to required memory movement to support execution at different depths.

[0208] Additionally, the chosen a value introduces an average 20 ms bias in latency estimation because of fixed point arithmetic and systematic noise. As depicted at line 17 of FIG. 33, to avoid premature reduction of depth due to this bias, it may be counteracted with a 20 ms bias in the PACE control engine.

[0209] Experimental Methodology. Intermittent Testing Setup. Intermittent TestingSetup. A ImF capacitor may be used for energy storage and similar to prior work, the P2110 Powerharvester Receiver may be utilized as the energy harvester. An experimental test-bed may be created to accurately simulate incoming energy and consistently replay a variety of volatile and non-volatile energy conditions. The incoming energy is simulated using a 3.2 voltage generated by a Digilent Analog Discovery 3 in series with a digital potentiometer. Using this setup, over 100 different volatile and non-volatile power traces may be generated. Non-volatile traces exhibit small energy fluctuations (2-5%), imitating real world harvesters. Volatile traces, on the other hand, randomly fluctuate in a wide range (20-80%) to imitate arbitrary changes to energy conditions (e.g. a person walking past an RF emitter or a cloud blocking the sun). Each experiment may be run on 10 randomly selected volatile and 10UCT-02225 randomly selected non-volatile traces (20 traces overall) and the average accuracy and percentage of deadline misses over all traces reported. The MSP430 clock is set to 16MHz.

[0210] Inference Applications. Three inference applications from the MLPerf tiny benchmark suit may be used. Prior work has used a subset of the applications or datasets from the MLPerf tiny benchmark suit. FIG. 32 lists applications used in the experiments. Some of the original networks outlined in the MLPerf benchmark suit require more model and activation memory than the MSP430 chip can support. The width of these oversized models may be modified to fit the device constraints. FIG. 32 lists applications used in the experiments. Some of the original networks outlined in the MLPerf benchmark suit require more model and activation memory than the MSP430 chip can support. We modify the width of these oversized models to fit the device constraints. FIG. 32 lists the width modification ratios. FIG. 32 also lists the required inference memory (activation + model weights) and percentage increase from the basic, nonadaptive variant of each application. The original network depths are not modified. In FIG. 32, DNN Application Details are from TinyML; inference latency is the latency of the most accurate tradeoff point; memory required for inference is expressed as both raw value (kB) and percentage increase from the basic, nonadaptive model. Unless otherwise noted, the latency constraint for each application is set to twice the latency of the most accurate tradeoff point, which is noted in FIG. 32.

[0211] Each application is trained as an Early Exit, Slimmable, and EarlySlim network (as explained above - Model Training). For proper comparison, the EarlySlim networks have the same number of switches as the Slimmables counterpart and same early exit location as the Early Exit counterpart. In general, Slimmable and EarlySlim models took *2 and *3.5 as long as Early Exit models to train.

[0212] The networks may be trained using Stochastic Gradient Descent (SGD) as the optimizer for 250-epochs, with a learning rate of 0.1, Nestrov momentum weight of 0.9, andUCT-02225 weight decay of 104on all non-BatchNorm parameters. A linear learning rate scheduler may be used, reducing the learning rate by 0.00039 each epoch. All models were 5 times on A40 GPUs using a batch size of 256. The average accuracy may be utilized in the results.

[0213] Points of Comparison. SLUMBER may be compared against prior adaptive inference runtimes as described in table 5. A comparison is not made against SLEXNet as it relies on a continuous power. For EE-, SL-, ES-Fixed, the recharge time lookup table may be populated as described in for a fixed deadline constraint (twice the inference latency noted in Table 6). Given these steps, EE-Fixed is a close approximation of ePerceptive. Tuned ES- Fixed is similar to ES-Fixed but individually tuned to support a variety of design-time latencies. EE-Greedy may be recreated to resemble Zygarde but without the energy availability predictor to compare against an adaptive approach that does not necessarily rely on design-time energy profiling. The work could not be recreated under the stochastic scheme.

[0214] Table 5Inference Mid-Inference Design-Time Used inName Model Strategy Profiling ExperimentsEE-Fixed Early Exit Fixed Once QI, Q2.2, Q4.2 (Coarsegrained)SL-Fixed Slimmable Fixed Once QI, Q2.2, Q4.2 (Rich)ES-Fixed Early Slim Fixed Once Q3.2 (Richer)Tuned ES-Fixed Early Slim Fixed Multiple Q3.2 (Richer)EE-Greedy Early Exit Greedy None QI, Q2 2 (CoarseProgressive grained)SLUMBER(ours) Early Slim Energy-Aware None All experiments (Richer) ProgressiveUCT-02225

[0215] PACE Configuration. PACE may be warmed up prior to experiments by running it under a variety of energy conditions for 10 minutes, sufficiently long enough for PACE to measure Tis for all the tradeoff points and for the EWMA lookup tables to converge. A lookup table of size 128 and a of 2'5may be used for the EWMA algorithm. Note that EWMA updates continue to occur throughout the experiments as long as inference is running. Except experiments described in Q3 : Design-Time Profiling, all PACE configurations operate with a 20 ms bias.

[0216] Performance Metrics. To compare, two performance metrics may be utilized: (1) accuracy and (2) percentage of deadline violations. For all points of comparison, average accuracy may be computed based on the number of timely inferences completed for each tradeoff point multiplied by the test accuracy of the underlying DNN model. Accuracy of a random guess may be substituted for inferences with missed deadlines. Higher accuracy reflects better performance. The percentage of inferences that missed their deadlines to a variety of reasons may also be reported. Lower missed deadlines are better. More timely approaches miss less.

[0217] Experimental Evaluation. Based on the experimental methodology discussed above (Experimental Methodology), the following questions (Qs) may be addressed,

[0218] QI - SLUMBER’S Overall Performance: Does SLUMBER improve accuracy and reduce deadline misses across a range of non-volatile and volatile energy conditions?

[0219] Q2 - Tradeoff Space Richness: Q2.1 Do EarlySlims provide a richer tradeoff space? Q2.2 How does a richer tradeoff space impact inference accuracy across a range of energy conditions?

[0220] Q3 - Design-Time Profiling: Q3.1 Can PACE’s latency estimation algorithm accurately, efficiently and dynamically predict the number of recharges needed for inferenceUCT-02225 tasks without design-time energy profiling? Q3.2 How does SLUMBER’S lack of design-time profiling compare against prior approaches?

[0221] Q4 - Mid-Inference Strategy: Q4.1 Can PACE’s control engine efficiently alter inference paths mid-execution? Q4.2 How does SLUMBER compare against prior approaches under volatile energy conditions? Does PACE act overly conservative or overly greedy?

[0222] QI: SLUMBER’S Improvement to Accuracy and Timeliness. 50 inference tasks may be conducted under 10 non-volatile and 10 volatile energy conditions (1000 inference in total) for each application using four different approaches. FIG. 26 illustrates the final results, specifically, Accuracy and Deadline Violations of three different applications (CIFAR-10, SPEECH, VWW) using four different Adaptive Inference approaches (from left to right in each graph: EE-Fixed, SL-Fixed, EE-Greedy, Slumber) on a variety of volatile and nonvolatile energy conditions, (a) Average Accuracy of each approach based on its respective DNN model (higher is better). The accuracy of inferences that missed their deadline may be replaced with a random guess. The percentage above each bar is the average inference accuracy for that approach. SLUMBER improves average accuracy by average of 15 percentage points, (b) Percentage of deadline misses for each approach (lower is better). The value above each bar is the percentage of deadline misses for that approach. SLUMBER reduces deadline misses by an average of 10 x.

[0223] Compared to prior Early Exit solutions, SLUMBER improves accuracy by an average of 15 percentage points and reduces deadline misses by an average of 10 x. These benefits are possible because of Early Slim’s rich tradeoff space and PACE’s energy-aware progressive mid-inference strategy, which are analyzed in more detail in the following subsections.UCT-02225

[0224] Q2: Tradeoff Space Richness. The inference tradeoff space may be analyzed in two stages. First, the tradeoff space of the discussed networks may be compared. Next, the impact of the tradeoff space on application performance may be evaluated.

[0225] Q2.1: Rich & Flexible Tradeoff Space. FIG. 27 compares the Pareto frontier tradeoff space of Early Exits 2701, Slimmables 2702, and Early Slims 2703 across the inference applications. These results may be compared based on three metrics: minimum energy threshold, fine-granularity, and overall, Pareto-frontier. The minimum energy threshold may be defined to be the least-energy consuming, yet better than random-guess tradeoff point of an adaptive DNN model. This minimum threshold delineates the smallest energy condition where DNN inference is feasible. A lower minimum threshold indicates a more flexible adaptive model that can operate in very low energy conditions. Hence, an inference model with high minimum threshold operating in low energy conditions is likely to miss the inference deadline. Since inference latency has a direct relationship with energy consumption, the models’ shortest, yet better than random-guess tradeoff point may be selected as the minimum energy threshold and compared against each other. As is shown in FIG. 27, EarlySlims have a lower minimum threshold than both Slimmables and Early Exits.

[0226] Various embodiments comprise evaluating fine-granularity based on the number of Pareto optimal tradeoff points. A fine-grained tradeoff space allows for a more fine-grained utilization of the available energy, improving inference accuracy and reducing energy underutilization. As illustrated in FIG. 27, Early Exit networks are the most coarse-grained. In contrast, both EarlySlims and Slimmables achieve equally comparable fine-granularity.

[0227] It was observed that EarlySlims and Slimmables have a better Pareto-frontier than Early Exits in CIFAR-10 and SPEECH applications. In VWW, EarlySlims perform comparably to Early Exits and better than Slimmables. Hence, the EarlySlim architecture and training do not noticeably degrade the inference accuracy in a continuous setting.UCT-02225

[0228] Q2.2: Accuracy Impact of Tradeoff Space. The richness of tradeoff space directly impacts application accuracy. First, A lower minimum energy threshold enables inference at more extreme energy conditions. As is shown in row b of FIG. 26, SLUMBER has no deadline misses in non-volatile energy conditions. This is due to EarlySlims very low minimum energy threshold, which enables SLUMBER to operate in extremely low energy conditions. In contrast, configurations using Early Exits (EE) and Slimmables (SL) have a higher minimum threshold and cannot operate in extremely low energy conditions. In such scenarios, both models run their cheapest tradeoff point but fail to complete the inference before the deadline, resulting in frequent deadline misses. The differences are visible across applications as well. In the CIFAR-10 application, EE-Fixed cannot run inference without violating the latency constraint in 3 out of 10 non-volatile energy conditions. On the other hand, EE-Fixed has a lower minimum energy threshold in the VWW application and fails to run inference in only 1 out of the 10 non-volatile energy conditions.

[0229] Secondly, fine-granularity improves the average inference accuracy. To compare the fine-granularity, the non-volatile energy experiments in FIG. 26 were repeated for energy conditions where the Early Exit approaches meet the deadline (i.e. no deadline misses occur due to high minimum energy thresholds). The results are shown in FIG. 28 which shows average accuracy under non-volatile power traces for three applications (CIFAR-10, SPEECH, VWW) each with four models (from left to right for each application: EE-Fixed, SL-Fixed, EE-Greedy, SLUMBER). This experiment was limited to traces that do not cause deadline misses in Early Exit and Slimmable approaches. The fine-granularity of EarlySlims and Slimmables allow for better utilization of available energy, improving the inference accuracy. On average, SLUMBER and SL-Fixed perform 3.5 percentage points better than the less granular Early Exit approaches. The difference in accuracy between SLUMBER and SL-Fixed stems from the minor difference in their Pareto-frontier, shown in FIG. 27.UCT-02225

[0230] Based on empirical results, it was found that a rich tradeoff space positively impacts application performance. Specifically, lower minimum energy thresholds of Early Slims enable SLUMBER to run inference in very low energy conditions without violating latency constraints and fine-granularity of Early Slims allow SLUMBER to better match inference accuracy to the variation in environmental energy conditions.

[0231] Q3: Design-Time Profiling. Various embodiments comprise analyzing the latency estimation algorithm in two stages. First, various embodiments comprise evaluating whether the algorithm can efficiently, accurately predict the number of required recharges and inference latency. Next, various embodiments comprise comparing SLUMBER’S lack of design-time profiling to prior work.

[0232] Q3.1: Efficient, Accurate Latency Prediction. In contrast to prior work,SLUMBER’S scheduler relies on real-time profiling of energy conditions. SLUMBER does so through its latency estimation algorithm. The efficiency and accuracy of this algorithm was evaluated (see, Latency Estimation Algorithm). As shown in Table 6, the latency estimation algorithm is both runtime and memory efficient. The runtime overhead arises from two sources: updating the EWMA lookup table at each recharge and computing the latency without any hardware division support. To measure the runtime overhead, 100 inference tasks were profiled across a range of non-volatile energy conditions and applications and report the average results in Table 6. The overhead of latency estimation algorithm is typically less than 2.5 ms, amounting to less than 0.12% of the total inference latency. The memory overhead of the estimator is also less than 0.8% of the total chip memory, with the main overhead stemming from the EWMA lookup table and the EWMA text region.

[0233] Table 6RuntimeMemory OverheadSource (kB)Overhead (ms)UCT-02225Latency Estimation <2.5 (0.12%) 2.0 (0.78%)Control Engine <7.5 (0.37%) 0.5 (0.19%)Overall Average <10 (0.49%) 2.6 (1%)

[0234] Similar to the experiment described above in Implementation, 100 CIFAR-10 inference tasks may be run with SLUBMER across a range of non-volatile energy conditions and the initial predicted latency, the initial number of predicated required recharges, real latency, and real number of required recharges recorded. Given the non-volatile nature of energy conditions, the accurate, initial estimates must be as close to real values as possible. Overall, SLUMBER’S estimates are fairly reliable, correctly estimating the number of needed recharges 93% of the time with a small bias of +21 ms when estimating the latency.

[0235] Q3.2: Design-Time Profiling-Free Prediction. The estimation algorithm requires no design-time profiling, saving developer time and avoiding profiling assumptions.

[0236] As discussed herein and noted in Table 7, prior inference approaches require extensive, time-consuming design-time profiling. SLUMBER, on the other hand, requires only modest deployment time profiling to warmup, as discussed in Implementation.

[0237] Table 7Design-Time Deployment-TimeSystemProfiling Duration Profiling Duration ePerspective Hours 0Zygarde Months 0SLUMBER 0 1-10 minutes (Once)

[0238] First, the estimation algorithm needs to know the continuous inference duration of each tradeoff point (TP). Depending on energy conditions, profiling this information can takeUCT-02225 anywhere between 1 and 10 minutes and will never require reprofiling. Second, the estimation algorithm starts with an unpopulated EWMA lookup table (all entries set to 1), which needs to be populated based on deployment time energy conditions. Based on the selected a and the discussion in Implementation, the lookup table and algorithm’s predictions should converge within 10 inference tasks. Prior to converging, the estimator only makes conservative predictions, resulting in timely results with suboptimal accuracy.

[0239] Additionally, solutions with design-time profiling are often optimal only under strict assumptions. For instance, the static recharge time lookup table employed by ePerceptive is inherently optimal for a single latency constraint. If the application designers were to change the latency constraint, they would need to reprofile the application for the new latency constraint. SLUMBER requires no such reprofiling.

[0240] To test this hypothesis, 100 CIFAR-10 inference tasks were run on SLUMBER and ES-Fixed across 10 non-volatile energy conditions. The experiments were repeated under 5 different latency constraints. Two versions of ES-Fixed were used. In the first version, the lookup table is tuned only to one latency constraint (0% deviation from design-time). In exemplary embodiments, such as Tuned ES-Fixed, the lookup table for each latency constraint is tuned. In some implementations, no steps are taken to tune SLUMBER. The experimental results are shown in FIG. 29 which shows CIFAR-10 average application accuracy across 10 non-volatile energy conditions and 5 different latency constraints. In FIG. 29, for each latency constraint each model shown left to right is: ES-Fixed, Tuned ES-Fixed, and SLUMBER. ES-Fixed is only tuned for one design-time latency, whereas Tuned ES- Fixed is tuned for all latency conditions. SLUMBER operates similarly to Tuned ES-Fixed without any additional design-time (re)profiling.

[0241] When deployment conditions deviate from design-time conditions, both Tuned ES- Fixed and SLUMBER operate better than ES-Fixed. For latency constraints smaller thanUCT-02225 design-time conditions, ES-Fixed schedules incorrectly and misses all the deadlines. For those above the design-time conditions, ES-Fixed acts conservatively and achieves subpar accuracy. Tuned ES-Fixed performs slightly better than SLUMBER due to the overestimation bias present in the latency estimation algorithm but requires additional profiling for each new deployment condition. It was observed that without any additional profiling, SLUMBER operates similarly to Tuned ES-Fixed, showcasing the strengths of SLUMBERS’ s deployment-agnostic scheduling.

[0242] Q4: Mid-Inference Strategy. Various embodiments comprise evaluating the performance of mid-inference strategies across points of comparison. Various embodiments comprise analyzing the efficiency of PACE’s control engine compared to prior adaptive midinference strategies (Greedy Progressive in EE-Greedy). Various embodiments comprise investigating the impact and performance of the energy-aware, progressive strategy under volatile and non-volatile energy conditions.

[0243] Q4.1 Efficient Mid-Inference Strategy. SLUMBER’S mid-inference strategy is more efficient than other non-fixed approaches. During non-volatile energy conditions in FIG. 26, EE-Greedy incurs so much additional runtime and energy overhead that the EE- fixed approach achieves higher accuracies. This impact is more pronounced in the CIFAR-10 application than the other applications because the CIFAR-lO’s early exit layers are quite expensive. In contrast, PACE’s energy-aware progressive strategy is quite efficient. It only incurs a maximum of 7.5 ms of runtime overhead (Table 6), which does not result in any accuracy reduction in various experiments.

[0244] Q4.2 Effective Mid-Inference Strategy. PACE’s mid-inference strategy performs better than both adaptive (EE-Greedy) and non-adaptive (EE- & SL-Fixed) strategies under volatile energy conditions. As shown in FIG. 26, SLUMBER incurs the lowest number of deadline violations during volatile energy conditions, an average of 3.8%, an order ofUCT-02225 magnitude above other approaches. SLUMBER’S success is due to its energy-aware progressive strategy, which intelligently alters the inference path when energy conditions deteriorate during inference. FIGS. 30A and 30B compares the behavior of SLUMBER to EE- & SL-Fixed under volatile and non-volatile energy conditions. FIGS. 30A and 30B show missed and successful inferences across EE-Fixed, SL-Fixed, and SLUMBER across 3 different applications and two different energy scenarios. In FIGS. 30A, missed is shown in the left side of each graph. For each application, EE-Fixed, SL-Fixed, SLUMBER Mid-inf Adapted and SLUMBER Initial Path are shown top to bottom for each application. Successes for SLUMBER are subdivided between SLUMBER Initial Path and SLUMBER Mid-inf Adapted. The initial label indicates the initially chosen tradeoff point was successful.SLUMBER Mid-inf Adapted indicates instances where the PACE altered the initial inference path to avoid a deadline violation. Overall, SLUMBER achieves lower deadline violations by adapting to energy conditions in volatile and non-volatile energy conditions.

[0245] In non-volatile, predictable energy conditions (FIG. 30 A), the initial latency estimates are mostly accurate and PACE almost never alters the inference path. In some embodiments, the path is occasionally altered when PACE overestimates the latency mid-inference. In contrast, PACE heavily alters the inference path during volatile, unreliable energy conditions to prevent deadline misses (FIG. 30B), preventing deadline misses that occur in the EE- & SL-Fixed configurations. Overall, it was observed that PACE intelligently alters the inference path when needed to ensure timely inference despite unknown, random fluctuations.

[0246] The real-time scheduling of adaptive, timely inference tasks for intermittent systems was studied. Various embodiments comprise identifying features and neural networks designs (EarlySlims) that improve that tradeoff space of adaptive DNN models and utilize Progressive Adaptive Control Engine (PACE) to schedule inference tasks despite unknown and volatile energy conditions, all without a priori profiling of design-time energyUCT-02225 conditions. An exemplary embodiment SLUMBER combines EarlySlims and PACE to improve system’s average accuracy by 15 percentage points while reducing deadline misses by an order of magnitude, paving the path for quick deployment of timely, accurate inference tasks on the intermittent edge.

[0247] The terms “computer system” and “computing device” are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on a type of computing system or computing device. In general, a computing system or computing device can be local or distributed and can include any combination of specialpurpose hardware and / or general-purpose hardware with software implementing the functionality described herein.

[0248] FIG. 31 illustrates a block diagram of hardware that may be employed in an implementation of the embodiments disclosed herein employing computer-executable instructions. FIG. 31 depicts a generalized example of a suitable general-purpose computing system 3100 in which the described innovations may be implemented in order to improve the processing speed and efficiency with which the computing system 3100 operates to perform the functions disclosed herein. With reference to FIG. 31 the computing system 3100 includes one or more processing units 3102, 3104 and memory 3106, 3108. The processing units 3102, 3106 execute computer-executable instructions. A processing unit can be a general-purpose central processing unit (CPU), processor in an application-specific integrated circuit (ASIC) or any other type of processor. The tangible memory 3106, 3108 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible by the processing unit(s). The hardware components in FIG. 31 may be standard hardware components, or alternatively, some embodiments may employ specialized hardware components to further increase the operating efficiency and speed with which the system 3100 operates. The variousUCT-02225 components of computing system 3100 may be rearranged in various embodiments, and some embodiments may not require nor include all of the above components, while other embodiments may include additional components, such as specialized processors and additional memory.

[0249] Computing system 3100 may have additional features such as for example, storage 3110, one or more input devices 3114, one or more output devices 3112, and one or more communication connections 3116. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 3100. Typically, operating system software (not shown) provides an operating system for other software executing in the computing system 3100, and coordinates activities of the components of the computing system 3100.

[0250] The tangible storage 3110 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way, and which can be accessed within the computing system 3100. The storage 3110 stores instructions for the software implementing one or more innovations described herein.

[0251] The input device(s) 3114 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 3100. For video encoding, the input device(s) 3114 may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing system 3100. The output device(s) 3112 may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system 3100.

[0252] The communication connect! on(s) 3116 enable communication over a communication medium to another computing entity. The communication medium conveys information suchUCT-02225 as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can use an electrical, optical, RF, or other carrier.

[0253] The terms “system” and “computing device” are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on a type of computing system or computing device. In general, a computing system or computing device can be local or distributed and can include any combination of special-purpose hardware and / or general-purpose hardware with software implementing the functionality described herein.

[0254] The foregoing explanation describes features of several embodiments so that those skilled in the art may better understand the scope of the invention. Those skilled in the art will appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and / or achieving the same advantages of the embodiments herein. Such equivalent constructions do not depart from the spirit and scope of the present disclosure. Numerous changes, substitutions and alterations may be made without departing from the spirit and scope of the present invention. While the above description contains many specificities, these should not be construed as limitations on the scope of the invention, but rather as an exemplification of preferred embodiments thereof Accordingly, the scope of the invention should be determined not by the embodiment(s) illustrated, but by the appended claims and their legal equivalents.

[0255] Although illustrative embodiments of the invention have been described in detail with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various changes and modifications can beUCT-02225 affected therein by one skilled in the art without departing from the scope of the invention as defined by the appended claims.

[0256] As shown in FIG. 34, computer system / server 12 in computing node 10 is shown in the form of a general -purpose computing device. The components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0257] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0258] Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system / server 12, and it includes both volatile and non-volatile media, removable and non-removable media.

[0259] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, aUCT-02225 magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

[0260] Program / utility 40, having a set (at least one) of program modules 42, may be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments as described herein.

[0261] Computer system / server 12 may also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (VO) interfaces 22. Still yet, computer system / server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer system / server 12 via bus 18. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with computer system / server 12. Examples, include, but are not limited to:UCT-02225 microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0262] The present disclosure may be embodied as a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0263] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD- ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiberoptic cable), or electrical signals transmitted through a wire.

[0264] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a localUCT-02225 area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0265] Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.UCT-02225

[0266] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0267] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0268] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0269] In various embodiments, a vector of features that includes machine learning model input(s) may be provided to one or more of the machine learning models described herein.UCT-02225Based on the input features, one or more of the machine learning models described herein may generate one or more outputs. In some embodiments, the output(s) of the one or more machine learning models described herein may be a vector of features.

[0270] A machine learning model is a computational system that can learn from data and perform tasks such as classification, regression, clustering, or generation. A machine learning model typically consists of a set of parameters that define its structure and behavior, and a training algorithm that adjusts these parameters based on a given objective function. In various embodiments, the one or more machine learning models, described herein, may be pre-trained using training data. In various embodiments, training data may be retrospective data. In various embodiments, the retrospective data may be stored in a datastore. In various embodiments, the one or more machine learning models, described herein, may be additionally trained through manual curation of previously generated outputs.

[0271] In various embodiments, the one or more machine learning models, described herein, may be and / or may include a dynamic programming algorithm and / or model, such as a dynamic linear programming algorithm / model or a dynamic nonlinear programming algorithm / model. In various embodiments, the one or more machine learning models, described herein, may be a trained classifier, a trained regression model, a clustering model, a generation model, and / or another type of model. In various embodiments, the trained classifier may be a random decision forest. However, it will be appreciated that a variety of other classifiers are suitable for use according to the present disclosure, including linear classifiers, support vector machines (SVM), or artificial neural network models, such as generative adversarial networks (GANs), a long short-term memory (LSTM) model, and / or recurrent neural networks (RNNs).

[0272] Suitable artificial neural network models include but are not limited to a feedforward neural network, a radial basis function network, a self-organizing map, learning vectorUCT-02225 quantization, a recurrent neural network, a Hopfield network, a Boltzmann machine, an echo state network, long short term memory, a bi-directional recurrent neural network, a hierarchical recurrent neural network, a stochastic neural network, a modular neural network, an associative neural network, a deep neural network, a deep belief network, a convolutional neural networks, a convolutional deep belief network, a large memory storage and retrieval neural network, a deep Boltzmann machine, a deep stacking network, a tensor deep stacking network, a spike and slab restricted Boltzmann machine, a compound hierarchical-deep model, a deep coding network, a multilayer kernel machine, a transformer, or a deep Q- network.

[0273] One or more machine learning models described herein may be trained in epochs and / or batches. An epoch is a complete pass through the entire training dataset. A batch is a subset of the training dataset that is processed at a time. For example, all batches may be processed during each epoch. A machine learning model may be updated after processing of each batch.

[0274] A machine learning model may be updated during training based on its loss and / or an optimization algorithm. Loss is determined using a loss function. A loss function is a mathematical expression that measures the discrepancy between the output of the model and the desired output, such as the ground truth labels or the target values. A loss function quantifies the error or the cost of the model’s predictions. A loss function provides a signal for the training algorithm to update the model’s parameters. Different loss functions may be suitable for different types of tasks and models and may have different properties and implications. Suitable loss functions include but are not limited to mean squared error (MSE), Cross-entropy, Kullback-Leibler divergence (KL divergence), contrastive, hinge, mean absolute error (MSE), Huber, Log-Cosh, and quantile loss.UCT-02225

[0275] An optimization algorithm is a numerical method that iteratively updates the model’s parameters based on the gradient of the loss function with respect to the parameters. An optimization algorithm aims to find the optimal or the best set of parameters that minimizes the loss function, or equivalently, maximizes the objective function. Suitable optimization algorithms include but are not limited to gradient descent, stochastic gradient descent (SGD), Adam, batch gradient descent, mini-batch gradient descent, momentum, Nesterov accelerated Gradient (NAG), adaptive gradient algorithm (Adagrad), and root mean squared propagation.

[0276] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0277] The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of theUCT-02225 embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0278]

Claims

UCT-02225WHAT IS CLAIMED IS:

1. A computer-implemented method for executing a neural-network computation, the method comprising: maintaining a set of control variables that encode progress through an artificial neural network; creating, at continuation points during execution, a checkpoint record comprising values of the control variables and a commit indicator; persisting the checkpoint record to a non-volatile buffer using an atomic logging scheme in which the commit indicator is written to indicate successful persistence; and upon restart after an interruption, reading the checkpoint record and resuming execution by restoring the control variables and continuing the neural -network computation from the encoded progress.

2. The method of claim 1, wherein the control variables correspond to indices of a plurality of nested loops that implement the neural -network computation, including one or more of: a layer index, a operator index, a tensor-tile index, a neuron index, or a time-step index.

3. The method of any one of claims 1-2, wherein the control variables comprise interlayer progress and intra-layer progress, the inter-layer progress indicating which layer or operation is active and the intra-layer progress indicating partial completion within the active layer or operation.

4. The method of any one of claims 1-3, wherein persisting the checkpoint record using the atomic logging scheme comprises writing the control variables to the non-volatile buffer and using available hardware support to detect or enforce that the write is committed.UCT-022255. The method of claim 4, wherein the commit indicator is encoded by setting a most- significant bit of at least one control variable as a commit bit, the commit bit being updated only after all control-variable values are durably written to the non-volatile buffer.

6. The method of any one of claims 4-5, wherein the available hardware support comprises transactional memory primitives that provide atomicity of the checkpoint record write.

7. The method of any one of claims 4-6, wherein the available hardware support comprises a read-modify-write atomic instruction to update the commit indicator.

8. The method of any one of claims 4-7, wherein the available hardware support comprises load-linked / store-conditional instructions to update the commit indicator conditionally on the absence of conflicting writes.

9. A computer-implemented method for performing inference using an artificial neural network model comprising: modifying width of the artificial neural network during inference whereby a first width is employed for a first layer and a second width is employed for a second layer.

10. The computer-implemented method of claim 9 further comprising: performing an early exit of the artificial neural network whereby one or more layers of the artificial neural network are not employed in generating an output of the inference.

11. The computer-implemented method of any one of claims 9-10, further comprising: accepting one or more inputs provided by an entity external to the artificial neural network to select depth and width of the artificial neural network in performing inference.

12. A computer-implemented method for training an artificial neural network model characterized by changeable width and depth during inference, the method comprising:UCT-02225 shrinking width of the artificial neural network at a rate, wherein a grouping of the rate is characterized by separate batch-norm statistics; and dropping an entire channel the artificial neural network when shrinking width; and training the artificial neural network model.

13. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable to perform the method of any one of claims 1-12.

Images