Systems and methods for efficient model execution on machine-learning accelerators

Abstract

A method, system, and device for reducing the latency of the configuration of an image sensor and an image signal processor. The computing system may include a machine learning (ML) processing engine capable of processing a denoising diffusion ML model for execution on a statically compiled ML accelerator. The system may determine that a partitioned graph representation comprising connected subgraphs representing at least one layer of the neural network of the ML model forms a directed acyclic graph. The system may insert cache nodes into the graph representation, wherein each cache node corresponds to each subgraph and is configured to cache the output of each subgraph. The system may generate an execution dataflow graph comprising multiple iterations of the partitioned graph representation, wherein during one or more iterations of a model inference operation, the execution dataflow graph uses inputs from cache nodes and excludes execution of the subgraphs corresponding to the cache nodes.

Description

Technology Field

[0001] This specification relates to neural networks, and more specifically, to a system and method for the efficient execution of a neural network model on a machine learning accelerator. Background Technology

[0002] Generative models are generally used to generate new data instances similar to the dataset on which the model is trained. For example, generative models can be used to generate images, text, audio, and video. Generative models learn the underlying patterns or distributions of data to generate new, similar data. One class of generative models includes diffusion models. Diffusion models add noise to the training data over multiple iterations. Then, the diffusion model is trained to reverse the noise process. In particular, diffusion models are trained to denoise the data step-by-step over multiple iterations, effectively reconstructing the original data from the noisy version. Once trained, the diffusion model can generate new data by starting with random noise and applying the learned denoising steps in reverse order. This process gradually transforms the noise into consistent and realistic data instances.

[0003] Diffusion models typically suffer from slow inference speeds. This is due to the iterative step-by-step denoising process, which cannot easily handle parallel computations. One approach to accelerating the execution of diffusion models is to use a caching method, where data, such as tensors generated in one iteration of the model, is cached and can be used in another iteration. In the iteration where cached data is used, parts of the model may be excluded from execution. This caching method requires conditional execution or flow to implement the inclusion or exclusion of parts of the model from execution. Many machine learning accelerators used on edge devices have simple instruction set architectures that enable the accelerator to run at low power on edge devices. Such simple instruction set architectures may not provide the conditional flow capability required by caching methods.

[0004] This specification describes a technique for executing a denoising diffusion machine learning model. In particular, the technique enables such a model to be executed on a statically compiled machine learning accelerator with increased efficiency and reduced power. The technique discussed herein may include processing a partitioned graph representation of a neural network implementing a machine learning model. The partitioned graph representation may include connected subgraphs, each subgraph including one or more layers of the neural network. Multiple cache nodes are inserted into the partitioned graph representation, wherein the cache nodes are configured to cache the outputs of the corresponding subgraphs. An execution dataflow graph may be generated using the partitioned graph representation, wherein the dataflow graph includes multiple iterations of the partitioned graph representation, and in one or more iterations, the execution dataflow graph uses inputs from one or more cache nodes instead of the corresponding subgraphs. As a result, one or more subgraphs are removed from execution. A compiler may compile each distinct subgraph in the execution dataflow graph individually and statically. At runtime, for each iteration, the machine learning accelerator can use executable code for each subgraph that is not excluded for execution in this iteration to run the model.

[0005] Specific examples of the subject matter of the invention described herein may be implemented to realize one or more of the following advantages. Generating an execution data flow graph that does not include any control flow or conditional execution enables the use of caching techniques for denoising diffusion machine learning models running on machine learning accelerators that do not support control flow or conditional execution. ML accelerators used on edge devices are designed for low power and have a simple instruction set architecture that may not support control flow or conditional execution. As a result, traditional caching techniques requiring control flow or conditional execution cannot be implemented on such accelerators. In contrast, the caching techniques discussed herein that do not require support for control flow or conditional execution can be easily implemented on machine learning accelerators. Thus, caching techniques for denoising noise in diffusion machine learning models can be implemented on edge devices.

[0006] Eliminating the execution of one or more parts of a model and instead using the cached output of those parts from previous iterations can reduce not only the computation time of the model execution but also the power consumption when running the model. The technique discussed herein can be applied to exclusion subgraphs that are redundant (i.e., whose outputs are not used) during iterations of the model execution. Consequently, the complex neural network computations associated with the exclusion parts are replaced instead by memory save and fetch operations associated with caching. Since the execution time and power consumption of these memory operations are substantially smaller than those for the exclusion parts, significant acceleration and power savings can be achieved over multiple iterations.

[0007] The user or user application can control the number of subgraphs excluded from execution by controlling splitting, as well as the frequency at which cache nodes provide cached data to another subgraph instead of the output of the excluded subgraph during iterative execution of the model. The user or user application may increase the frequency to reduce power consumption and execution time, but at the expense of reduced output quality of the model. Nevertheless, the technique discussed herein provides the user or user application with the ability to control power consumption and execution time versus the output quality of the model when executed over multiple iterations.

[0008] One general embodiment includes a method for executing a denoising diffusion machine learning model on one or more machine learning accelerators. The method comprises the steps of: determining by a computing device that a partitioned graph representation of a neural network forms a directed acyclic graph—the partitioned graph representation comprises connected subgraphs each representing at least one layer of the neural network—; inserting one or more cache nodes into the partitioned graph representation by the computing device—each cache node corresponds to a respective subgraph of the connected subgraphs and is configured to cache the output of each subgraph during model execution—; generating an execution dataflow graph comprising a plurality of iterations of the partitioned graph representation by the computing device—in one or more iterations during model inference operations, the execution dataflow graph uses inputs from one or more cache nodes and excludes execution of the subgraphs corresponding to one or more cache nodes—; and compiling each distinct subgraph in the execution dataflow graph individually by the computing device to generate runtime code for execution on one or more machine learning accelerators.

[0009] An embodiment may include one or more of the following features. In the method, the output of each subgraph during model execution is an output tensor. For each iteration of a plurality of iterations, the execution dataflow graph excludes the execution subgraph from being fed to a cache node that feeds to another subgraph. For an iteration, the execution dataflow graph excludes the execution of the subgraph by marking the subgraph so that it is not executed during the runtime for this iteration. One or more machine learning accelerators execute the plurality of iterations during a model inference operation based on code compiled individually for each subgraph. The method may include the step of determining the frequency of iterations in which a cache node corresponding to a specific subgraph provides an input to a subsequent subgraph; and the step of determining whether the cache node will provide an input to the subsequent subgraph based on the frequency of iterations. The method, based on the frequency of iterations: in a first iteration, the cache node determines that a first input will be provided to the subsequent subgraph and that the specific subgraph will be excluded during model execution; in a second iteration, the step of determining that the specific subgraph will provide a second input to the cache node and the subsequent subgraph, respectively; And in the third iteration, the method may further include the step of determining that a specific subgraph provides a third input to a subsequent subgraph and that a cache node will be excluded during model execution. The partitioned graph representation is a first partitioned graph representation, and the denoising diffusion machine learning model is a first denoising diffusion machine learning model; and the method may include the step of determining that a second partitioned graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a directed acyclic graph; and the step of stopping the execution of the second denoising diffusion machine learning model.

[0010] One general embodiment includes one or more non-transient computer-readable storage media storing instructions that, when executed by one or more processors, cause one or more processors to perform an operation for executing a denoising diffusion machine learning model on one or more machine learning accelerators. The operation includes an operation by a computing device determining that a partitioned graph representation of a neural network forms a directed acyclic graph—the partitioned graph representation comprises connected subgraphs each representing at least one layer of the neural network—; an operation by the computing device inserting one or more cache nodes into the partitioned graph representation—each cache node corresponds to a respective subgraph of the connected subgraphs and is configured to cache the output of each subgraph during model execution—; an operation by the computing device generating an execution dataflow graph comprising multiple iterations of the partitioned graph representation—in one or more iterations during model inference operation, the execution dataflow graph uses inputs from one or more cache nodes and excludes execution of the subgraphs corresponding to one or more cache nodes—; It includes the operation of compiling each distinct subgraph in an execution data flow graph individually to generate runtime code for execution on one or more machine learning accelerators by a computing device.

[0011] In a computer-readable storage medium, the output of each subgraph during model execution is an output tensor. For each iteration of a plurality of iterations, the execution dataflow graph excludes the execution subgraph from being fed to a cache node that feeds to another subgraph. For an iteration, the execution dataflow graph excludes the execution of the subgraph by marking the subgraph so that it is not executed during the runtime for this iteration. One or more machine learning accelerators execute the plurality of iterations during a model inference operation based on code compiled individually for each subgraph. The operation may include an operation to determine the frequency of iterations in which a cache node corresponding to a specific subgraph provides an input to a subsequent subgraph; and an operation to determine whether the cache node provides an input to the subsequent subgraph based on the frequency of iterations. The operation, based on the frequency of iterations: an operation to determine, in a first iteration, that the cache node provides a first input to the subsequent subgraph and that the specific subgraph will be excluded during model execution; an operation to determine, in a second iteration, that the specific subgraph will provide a second input to the cache node and the subsequent subgraph, respectively; And in the third iteration, the operation may include determining that a specific subgraph provides a third input to a subsequent subgraph and that a cache node will be excluded during model execution. The partitioned graph representation is a first partitioned graph representation, and the denoising diffusion machine learning model is a first denoising diffusion machine learning model; the operation may include determining that a second partitioned graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a directed acyclic graph; and stopping the execution of the second denoising diffusion machine learning model.

[0012] One general embodiment includes a system. The system also includes one or more processors; and one or more storage devices that store instructions for performing operations to execute a denoising diffusion machine learning model on one or more machine learning accelerators when executed by one or more processors, wherein the denoising diffusion machine learning model is implemented using a neural network, and the operations include: an operation by a computing device to determine that a partitioned graph representation of the neural network forms a directed acyclic graph—the partitioned graph representation includes connected subgraphs each representing at least one layer of the neural network—; an operation by a computing device to insert one or more cache nodes into the partitioned graph representation—each cache node corresponds to a respective subgraph of the connected subgraphs and is configured to cache the output of each subgraph during model execution—; an operation by a computing device to generate an execution dataflow graph comprising a plurality of iterations of the partitioned graph representation—in one or more iterations during model inference operations, the execution dataflow graph uses inputs from one or more cache nodes and excludes execution of the subgraphs corresponding to one or more cache nodes—; It may include the operation of individually compiling each distinct subgraph in an execution data flow graph to generate runtime code for execution on one or more machine learning accelerators by a computing device.

[0013] In the system, the output of each subgraph during model execution is an output tensor. For each iteration of multiple iterations, the execution dataflow graph excludes the execution subgraph from being fed to a cache node that feeds to another subgraph. For an iteration, the execution dataflow graph excludes the execution of the subgraph by marking the subgraph so that it is not executed during the runtime for that iteration. One or more machine learning accelerators execute multiple iterations during a model inference operation based on code compiled individually for each subgraph. The operation may include an operation determining the frequency of iterations in which a cache node corresponding to a specific subgraph provides an input to a subsequent subgraph; and an operation determining whether the cache node provides an input to the subsequent subgraph based on the frequency of iterations. The operation, based on the frequency of iterations: an operation determining that, in a first iteration, the cache node provides a first input to the subsequent subgraph and that the specific subgraph will be excluded during model execution; and an operation determining that, in a second iteration, the specific subgraph will provide a second input to the cache node and the subsequent subgraph, respectively. And in the third iteration, the operation may include determining that a specific subgraph provides a third input to a subsequent subgraph and that a cache node will be excluded during model execution. The partitioned graph representation is a first partitioned graph representation, and the denoising diffusion machine learning model is a first denoising diffusion machine learning model; the operation may include determining that a second partitioned graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a directed acyclic graph; and stopping the execution of the second denoising diffusion machine learning model.

[0014] Details regarding one or more embodiments of the subject matter of the invention of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Brief explanation of the drawing

[0015] Figure 1 is a block diagram of an exemplary computing system. Figure 2 illustrates a block diagram of an exemplary ML model processing engine. Figure 3a illustrates the UNet and UViT high-level model architectures of the denoising diffusion machine learning model. FIG. 3b illustrates a first exemplary partition of the model shown in FIG. 3a, as well as a first exemplary partitioned graph representation of a neural network implementing the denoising diffusion machine learning model shown in FIG. 3a. FIG. 3c illustrates a second exemplary partition of the model shown in FIG. 3a, as well as a second exemplary partitioned graph representation of a neural network implementing the denoising diffusion machine learning model shown in FIG. 3a. Figure 4a illustrates a partitioned graph representation having cache nodes. FIG. 4b illustrates an exemplary execution data flow graph based on the segmented graph representation shown in FIG. 4a. Figure 5a illustrates the high-level model architecture of the Diffusion Transformer (DiT) of a denoising diffusion machine learning model. FIG. 5b illustrates the division of the exemplary model shown in FIG. 5a. Figure 6a illustrates a partitioned graph representation having cache nodes. FIG. 6b illustrates an exemplary execution data flow graph based on the segmented graph representation shown in FIG. 6a. Figure 7 illustrates an exemplary process flowchart. In various drawings, similar reference numbers and names represent similar elements. Specific details for implementing the invention

[0016] FIG. 1 is a block diagram of an exemplary computing system (100). The exemplary computing system (100) may include a system-on-chip (102) (“SoC (102)”), which may again include a central processing unit (CPU) (104), shared memory (108), and IP / circuit blocks (110). The SoC (102) may be implemented in an integrated circuit of an exemplary user / client device (130), consumer device, or mobile device, each of which may include items such as a smartphone (130a), tablet (130b), laptop (130c), and smartwatch or wearable device (130d). The client device (130) may include other items such as an electronic notebook, netbook, smart speaker, or mobile computer. In some examples, the system (100) and the SoC (102) are integrated circuits of a desktop computer, network server, or related cloud-based asset. The SoC (102) may be combined with an image sensor (152). FIG. 1 illustrates a single image sensor (152), but more than one image sensor may be included. In particular, the image sensor (152) may be part of a client device (130).

[0017] The CPU (140) may be a general-purpose CPU (e.g., a single or multi-core CPU). An operating system may run on the CPU (140). Examples of operating systems may include an Android operating system, an iOS operating system, a Windows operating system, etc. The operating system may run one or more image processing applications. For example, image processing applications may include photo applications, video calling applications, etc. The operating system may also run generative imaging applications such as text-image applications, super-resolution applications, and outpainting applications. Generative imaging applications may utilize machine learning accelerators on the SoC (102) as well as the image processing stack of the operating system.

[0018] The CPU (104) may execute an ML model processing engine (142). The ML model processing engine (142) may be executed as an application on the operating system of the CPU (104) or may be native to the operating system. The ML processing engine (142) may be stored in memory (108) and may be loaded into the CPU (104) local memory during execution. In some examples, at least part of the ML model processing engine (142) (e.g., a compiler) may be executed on the SoC (102) or on another processor outside the SoC (102). In such cases, the ML processing engine (142) may communicate data with the other processor. The ML model processing engine (142), which is discussed in detail below, allows for modification of the execution of the diffusion model by inserting cache nodes in a manner that allows it to be executed on one or more statically compiled machine learning accelerators.

[0019] Memory (108) is system memory, shared memory, or both. In the example of FIG. 1, memory (108) is depicted outside the circuit block (110). However, memory (108) may include a portion of memory that is i) specific to the circuit block (110), ii) outside the circuit block (110), or iii) both. Memory (106) may be random access memory of the SoC (102), such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), or dual data rate (DDR) SDRAM. In some embodiments, an aspect of memory (108) is configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block (110). The memory (108) may include various other types of memory, such as high-bandwidth memory (HBM), narrow-band memory (for example, to store 8-bit values), and wide-band memory (for example, to store 16-bit or 32-bit values).

[0020] The circuit block (110) may include an image signal processor (ISP) (112), a tensor processing unit (TPU) (114), a digital signal processor (DSP) (116), and a graphics processing unit (GPU) (118). The circuit block (110) may also be referred to as an IP block (110), wherein the IP block may include one or more proprietary hardware elements. For example, each of the ISP (112), TPU (114), DSP (116), and GPU (118) may be a proprietary IP block (or IP device) of a specific entity or device manufacturer. The TPU (114), DSP (116), and GPU (118) may be considered as AI accelerators capable of efficiently processing AI workloads such as neural networks. Applications running on the CPU (104) may offload AI-based computations to the TPU (114), DSP (116), and / or GPU (118).

[0021] Some AI accelerators, particularly those designed for low-power operation, do not provide control flow or dynamic conditional execution features. Removing conditional execution features from AI accelerators allows for simpler instruction set architectures that can be optimized for performance and power. This enables AI accelerators to be integrated into edge devices, such as battery-powered mobile devices. Nevertheless, the lack of conditional execution capabilities during runtime makes AI accelerators unusable for traditional caching technologies that require conditional execution. As discussed below, ML model processing engines modify the graph representation of neural networks to include cache nodes without the need for conditional execution.

[0022] FIG. 2 illustrates a block diagram of an exemplary ML model processing engine (142). The ML model processing engine (142) may include a data flow graph processing engine (202), a scheduler (204), a compiler (206), and an allocation engine (208). In some examples, the data flow graph processing engine (202) may receive a segmented graph representation (210) of a neural network implementing a machine learning model, such as a denoising diffusion machine learning model. A user may provide the segmented graph representation (210) to the ML model processing engine (142). The user may include an application developer, an ML practitioner, etc. In some examples, the user may store the segmented graph representation (210) in memory and provide the memory address where the segmented graph representation (210) is stored to the ML model processing engine (142). In some examples, the segmented graph representation (210) may be saved as a TensorFlow SavedModel format containing a complete model with trained parameters and computations. In some examples, the segmented data flow graph representation (210) may be in the form of code or description, such as a TensorFlow computation graph description, or a computation graph description in other machine learning platforms such as JAX, TensorFlow Lite, Pytorch, etc.

[0023] The data flow graph processing engine (202) can process the partitioned graph representation to determine whether the graph forms a directed acyclic graph (DAG). In some examples, the data flow graph processing engine (202) can identify the nodes and edges of the partitioned graph representation (201) and then check for the presence of cycles in the graph. In some cases, a depth-first search traversal of the graph may be performed. During the traversal, the data flow graph processing engine (202) can track the nodes encountered. If a previously encountered node is traversed again, it may be determined that a cycle exists in the graph and the graph is not a DAG. In other cases, other techniques, such as Kahn's algorithm or other topology classification algorithms, may be used. If the data flow graph processing engine (202) determines that the partitioned graph representation is indeed a DAG, processing may continue. However, if the data flow graph processing engine (202) determines that the partitioned graph representation does not form a DAG, processing of the graph is terminated. In some examples, the ML model processing engine (142) may report to the user or user application that the segmented graph representation does not form a DAG.

[0024] The scheduler (204) may insert one or more cache nodes into the partitioned graph representation. A cache node represents a caching operation in which the output of a subgraph of the partitioned graph representation corresponding to the cache node is stored in the cache. The stored output of a specific cache node may be used instead of the output of this subgraph in subsequent iterations of the execution of the partitioned graph representation. The scheduler (204) may generate an execution data flow graph comprising multiple iterations of the partitioned graph representation. The execution data flow graph may be represented in a manner similar to the partitioned graph representation. For example, the scheduler (204) may generate the execution data flow graph using a TensorFlow computation graph description. The ML model processing engine (142) determines the placement of the cache nodes and the frequency at which data within the cache is provided as input to the subgraph during execution (e.g., every n Information regarding the (th iteration) can be received as input. For example, the input information may specify placing a cache node between two subgraphs and using cached data as input every 3rd iteration. The scheduler (204) may exclude one or more subgraphs from execution if the subgraphs are duplicates, taking into account the use of cached data. The scheduler (204) generates an execution data flow graph in such a way that the graph does not result in arbitrary conditional execution. That is, the scheduler (204) generates an execution data flow graph without arbitrary conditional execution by excluding the execution of duplicate subgraphs.

[0025] The scheduler (204) may provide the execution data flow graph to the compiler (206). The compiler (206) may be configured to convert the execution data flow graph into runtime code for execution on one or more statically compiled machine learning accelerators. As mentioned above, the execution data flow graph may be a computation graph format, such as a TensorFlow computation graph description. The compiler (206) may extract a compilable representation for each subgraph and statically compile each subgraph individually. This removes all dynamic properties during the compilation phase. Instead, dynamic properties are shifted to application logic at runtime, which determines whether to execute the subgraph in each iteration. The compiler (206) may optimize the runtime code for specific hardware accelerators and underlying memory architectures. For example, the compiler (206) may optimize the runtime code for memory allocation, hiding memory latency, parallelization, etc. Examples of compilers that can be used to implement the compiler (206) may include XLA, Glow, Apache TVM, etc.

[0026] The allocation engine (208) may allocate hardware resources for the execution of runtime code. For example, the allocation engine (208) may allocate one or more machine learning accelerators (212) to execute runtime code. In some examples, the allocation engine (208) may also allocate memory resources to store weights and the outputs of various layers of a neural network executed by the runtime code. Hardware accelerators (212) may include TPUs, GPUs, neural processing units (NPUs), etc. For example, one or more of NVIDIA GPUs (e.g., A100 series), Google’s TPUs, Graphcore’s Intelligent Processing Units (IPUs), etc., may be used to implement one or more hardware accelerators (212).

[0027] As mentioned above, the ML model processing engine (142) can receive a segmented graph representation of a neural network implementing a denoising diffusion machine learning model. The segmented graph representation may be based on one or more neural network model architectures, such as UNeT, UViT, or DiT.

[0028] FIG. 3a illustrates the UNeT and UViT high-level model architecture of a denoising diffusion machine learning model (300) ("DM(300)"). The DM (300) includes a set of encoder stages (302), an intermediate stage (304), and a set of decoder stages (306). The DM (300) receives an input (308), which may be data such as an image, and adds noise in each of the encoder stages (302). The intermediate stage (304) may include residual neural network blocks. The set of decoder stages (306) is trained to regenerate an image from the noisy image generated by the set of encoder stages (302). The output (310) of the set of decoder stages (306) is the regenerated image.

[0029] FIG. 3b illustrates a first exemplary partition of the model illustrated in FIG. 3a, as well as a first exemplary partitioned graph representation of a neural network implementing the denoising diffusion machine learning model illustrated in FIG. 3a. The first exemplary partition (350) includes two partitions (P0 and P1). Partition (P0) includes encoding steps (D1 and D2) as well as decoding steps (U1 and U2). Partition (P2) includes the remainder of the model (300). Additionally, FIG. 3b illustrates a partitioned graph representation (352) of a neural network implementing the model (300) illustrated in FIG. 3a. The partitioned graph representation (352) includes partitions (P0 and P1) illustrated as nodes of the graph, and an edge between P0 and P1 representing data flow between the partitioned models. The partitions (P0 and P1) may represent connected subgraphs (P0 and P1), where each subgraph may represent at least one layer of the neural network. In this example, subgraph (P0) represents the neural network layers of the encoding steps (D1 and D2) as well as the decoding steps (U1 and U2), while subgraph (P1) represents the remaining neural network layers of the model (300). As mentioned above, the partitioned graph representation (352) may be described in a computational graph description, such as a TensorFlow computational graph description. In some examples, a user or user application may provide the data flow graph processing engine (202) with a computational graph description of the partitioned graph representation (352) as well as a computational graph description of each subgraph. In some examples, a user or user application may provide the data flow graph processing engine (202) with a computational graph description of a complete high-level model (e.g., DM (300) shown in FIG. 3a) and partition boundaries representing nodes of the model that are grouped together into subgraphs. In such cases, the data flow graph processing engine (202) may generate a partitioned graph representation (352) from the high-level model and partition boundaries.

[0030] In some examples, the data flow graph processing engine (202) may generate partition boundaries itself and then generate a partitioned graph representation based on the graph representation of the model and the partition boundaries. Before the inference operation, the data flow graph processing engine (202) generates several partition boundaries and the output of the model (e.g., an image generated by the model that is larger than a threshold) Power savings and execution time reductions resulting from each of the multiple partition boundaries can be determined while maintaining quality metrics (such as the Inception Distance (FID) metric). The data flow graph processing engine (202) can provide the user or user application with the generated partition boundaries as well as the corresponding power, efficiency, and quality metrics. Then, the user or user application can select one of the partition boundaries to partition the model for inference operations.

[0031] Referring to FIG. 2, the data flow graph processing engine (202) can determine whether the partitioned graph representation (352) illustrated in FIG. 3b is a DAG. A DAG is a type of graph in which nodes are directionally related to each other and do not form a directed closed loop. The data flow graph processing engine (202) can determine that the partitioned graph representation (352) does not form a DAG. This is because data flows from subgraph (P0) to subgraph (P1) and back from subgraph (P1) to subgraph (P0), thereby forming a directed closed loop. As a result, the data flow graph processing engine (202) can terminate the processing of the graph representation and convey to the application that the partitioned graph representation is not a DAG. It should be noted that subgraphs are executed atomically. A partitioned graph representation that is not a DAG may generate circular dependencies that do not align with the atomical execution of the subgraphs. Accordingly, the data flow graph processing engine (202) ensures that the divided graph representation is a DAG before performing additional processing.

[0032] FIG. 3c illustrates a second exemplary partition (360) of the model illustrated in FIG. 3a, as well as a second exemplary partitioned graph representation (362) of a neural network implementing the denoising diffusion machine learning model illustrated in FIG. 3a. The second exemplary partition includes three partitions (P0, P1, and P2). Partition (P0) includes encoding steps (D1 and D2), Partition (P1) includes decoding steps (U1 and U2), and Partition (P2) includes the remainder of the model (300). Additionally, the partitions (P0, P1, and P2) may represent connected subgraphs (P0, P1, and P2), where each subgraph includes at least one layer of the neural network implementing the diffusion model (300). FIG. 3c also illustrates a second exemplary partitioned graph representation based on the subgraphs (P0, P1, and P2). A second exemplary segmented graph representation (362) includes subgraphs (P0, P1 and P2) depicted as nodes of a graph, and edges between subgraphs (P0, P1 and P2) representing data flow between segmented models.

[0033] Referring again to FIG. 2, the data flow graph processing engine (202) can determine that the second exemplary partitioned graph representation (362) is actually a DAG because the graph does not form a directed closed loop. As a result, the data flow graph processing engine (202) can provide the graph to the scheduler (204). As discussed in relation to FIG. 2, the scheduler (204) can insert one or more cache nodes into the partitioned graph representation.

[0034] FIG. 4a illustrates a partitioned graph representation (462) having a cache node (402). An ML model processing engine (142) may receive the location of the cache node in the partitioned graph representation as input. In the example illustrated in FIG. 4a, the ML model processing engine (142) receives an input instructing it to insert a cache node corresponding to a subgraph (P1). As illustrated in FIG. 4a, the scheduler (204) may insert a cache node (C) that receives the output of the subgraph (P1). The cache node may represent a memory operation in which the output of the subgraph (P1) is stored in memory and may be retrieved at a later stage during the execution of the model. For example, the memory operation may store the output of the subgraph (P1) in system memory (108) (Fig. 1), local memory of the CPU (104), or local memory of one or more of the ML accelerators. The ML model processing engine (142) may receive as input a command to insert a cache node (C) between a corresponding subgraph (P1) and another adjacent subgraph (P2). In effect, the cache node (C) is inserted between two adjacent nodes and parallel to the existing edge between the two adjacent nodes. The scheduler (204) may include an edge pointing from the first node (subgraph (P1)) to the cache node among two adjacent nodes indicating that the output of the subgraph (P1) is stored in the cache node (C). The scheduler (204) may include a second edge pointing from the cache node to the second node (subgraph (P2)) among two adjacent nodes indicating that the output stored in memory is provided as input to the subgraph (P2). The scheduler (204) may generate an execution data flow graph comprising multiple iterations of the partitioned graph representation based on the partitioned graph representation.

[0035] FIG. 4b illustrates an exemplary visual representation of an exemplary execution data flow graph (450) based on the partitioned graph representation shown in FIG. 4a. A scheduler (402) can generate the execution data flow graph (450) by linearly arranging the repetitions of the execution of the partitioned graph representation (462). A portion of the execution data flow graph (452) shown in FIG. 4b includes four repetitions, namely repetition 0, repetition 1, repetition 2, and repetition 3. In one repetition, the output of the subgraph (P2) is fed as an input to the subgraph (P0) of the next repetition.

[0036] The number of iterations in which a partitioned graph representation is executed can be predetermined. A scheduler (204) can process the partitioned graph representation to determine whether any subgraph should be excluded from execution for each iteration. The scheduler (204) may also consider the requested location and frequency (also referred to as "cache input insertion frequency") in which data stored in a cache node is used as input for a subgraph. In the example illustrated in FIGS. 4a and 4b, the cache node is positioned corresponding to a subgraph (P1), so that the cache node is connected in parallel between the subgraphs (P1 and P2), and the cached data is used as input once every three iterations, starting from iteration 2. Of course, this is merely an example, and the cache input insertion frequency in which cached data is used as input can vary. Generally, increasing the cache input insertion frequency can reduce the execution time of the model, but it also reduces the accuracy of the model. Thus, there is a trade-off between execution time and the quality of the model's output.

[0037] The scheduler (204) may determine whether the cache node provides input to the subsequent subgraph for each iteration based on the cache input insertion frequency. For example, based on the cache input insertion frequency for every three iterations starting from iteration 2, the scheduler (204) may determine that the cache node should provide input to the subgraph (P2) for every three iterations starting from iteration 2. Thus, in iteration 2, the cache node (C) provides input to the subgraph (P2). For the iteration in which the cache node provides input to the consuming subgraph, the scheduler (204) may determine that the producer subgraph, which provided output to the cache node in the previous iteration, will be excluded from execution. For example, in iteration 2, the cache node provides input to the subgraph (P2), and the input stored in the cache node (C) is provided by the producer subgraph (P1) in the previous iteration 1. Accordingly, in iteration 2, the scheduler (204) may mark the subgraph (P1) for non-execution. The scheduler (204) may also determine that for another iteration, the generating subgraph will provide input to both the cache node and the consuming subgraph. For example, the scheduler (204) may determine that the subgraph (P1) provides input (i.e., the output of the subgraph (P1)) to both the cache node and the consuming subgraph (P2). Additionally, the scheduler (204) may determine that the cache node is not executed for an iteration in which the generating subgraph provides input to the consuming subgraph. For example, in iteration 0, the subgraph (P1) provides input to the subgraph (P2), and the cache node is not executed.

[0038] Generally, the scheduler (204) can determine whether the cache node provides input to the consuming subgraph based on the cache input insertion frequency for each iteration. If the cache node provides input to the consuming subgraph in an iteration, the cache node is executed to provide input to the consuming subgraph, and the subgraph that generated the input stored in the cache is excluded from execution (for example, subgraph (P1) is excluded from execution in iteration 2 because subgraph (P1) generated the input stored in the cache in iteration 1). For a first iteration in which the cache node does not provide input to the consuming subgraph, the scheduler (204) determines whether the cache node provides input to the consuming subgraph in the next iteration. If the cache node provides input to the consuming subgraph in the next iteration, the cache node consumes the output from the producing subgraph in the first iteration (for example, in iteration 1, the cache node (C) does not provide input to the consuming subgraph (P2), and in the next iteration, iteration 2, the cache node provides input to the consuming subgraph (P2). In this case, in iteration 1, the cache node is executed to store the output of the producing subgraph (P1). Additionally, for the first iteration in which the cache node does not provide input to the consuming subgraph, the scheduler (204) determines whether the cache node does not provide input to the consuming subgraph in the next iteration. If the cache node does not provide input to the consuming subgraph in the next iteration, the cache node is excluded from execution in the first iteration (for example, for iteration 0 in which the cache node does not provide input to the consuming subgraph, the scheduler (204) may determine that the cache node (C) does not provide input to the consuming subgraph (P2) in the next iteration, iteration 1. Thus, the execution of the cache node is excluded in iteration 0). Additionally, the scheduler (204) may determine the subgraph in the iteration that feeds only to the subgraph excluded in the next iteration.Since only the subgraph of the next iteration is excluded, a subgraph that feeds only to the subgraph excluded in the previous iteration may be excluded. The scheduler (204) can determine whether a node or subgraph should be marked as non-executable using a data flow liveness / accessibility analysis algorithm. For example, algorithms such as dead-code deletion, dead-code removal, and dead-code stripping can be used to determine which subgraphs are duplicated and can be removed from execution.

[0039] It should be noted that the execution of the cache node in an iteration may differ based on whether the cache node (C) is consuming or providing input. For example, in iteration 1, where the cache node (C) consumes the output of the subgraph (P1), the cache node is executed to store the output of the subgraph (P1) in memory. In contrast, in iteration 2, where the cache node (C) provides input to the subgraph (P2), the cache node is executed to retrieve the output of the subgraph (P1) stored in memory and provide it as input to the subgraph (P2).

[0040] As is evident from the execution data flow graph of FIG. 4b, the execution of subgraph (P1) begins at iteration 2 and is removed at every 3rd iteration and replaced by two memory operations: one in the previous iteration stores the output of subgraph (P1) in memory, and in the same iteration, the second retrieves the output from memory and provides it as input to subgraph (P2). That is, the execution of complex neural network computations is substantially replaced by low-latency and low-power memory storage and fetch operations. As a result, over multiple iterations of the model, both execution time and power consumption can be reduced.

[0041] Referring again to FIG. 2, the scheduler (204) may provide the execution data flow graph to the compiler (206) for runtime code generation. The compiler (206) may statically compile each subgraph of the execution data flow graph individually. For example, the compiler (206) may process the execution data flow graph (452) illustrated in FIG. 4b and statically compile each subgraph (P0, P1, P2) and C individually. Since each subgraph is statically compiled, there is no need to support control flow or conditional execution from the ML accelerator. The dynamic characteristic determining whether a subgraph will be executed is made during runtime based on whether the subgraph is marked as non-executable. By compiling each subgraph individually, the application logic at runtime may use the code compiled individually for each subgraph to schedule the execution of each subgraph during each iteration of the model's execution. Therefore, a caching method to reduce execution time and power consumption during the execution of a denoising diffusion ML model can be implemented on an ML accelerator used on an edge device.

[0042] It should be noted that the ML model processing engine (142) can process any graph representation of the model. In the examples illustrated in FIGS. 3a through 4b, the model is represented by UNeT and UViT high-level model architectures. However, other architectural representations may also be processed.

[0043] FIG. 5a illustrates a high-level model architecture of a Diffusion Transformer (DiT) of a denoising diffusion machine learning model (500), and FIG. 5b illustrates a partition of the exemplary model illustrated in FIG. 5a. A DiT is a class of diffusion models based on a transformer architecture. A DiT can provide improved performance by replacing a commonly used UNet convolutional neural network backbone with a transformer. The DiT model (500) illustrated in FIG. 5a may include a series of DiT blocks (D1, D2, D3...Dn), which form the denoising portion of an inference operation using the DiT. The input to block (D1) is noise, and the output of block (Dn) is transformed data such as an image. FIG. 5b illustrates an exemplary partition (550) in which partition (P1) includes blocks (D1 and D2), while partition (P2) includes the remainder of the model's blocks. This is an exemplary partition, and it should be understood that various implementations may partition the model differently. FIG. 5b also illustrates a segmented graph representation (562) in which the output of a subgraph (P0) is provided as an input to a subgraph (P1).

[0044] The data flow graph processing engine (202) can determine whether the partitioned graph representation (562) illustrated in FIG. 5b forms a DAG. In this case, the partitioned graph representation (562) does not form any closed loop. Therefore, the data flow graph processing engine (202) can determine that the partitioned graph representation (562) can be further processed. The ML model processing engine (142) can receive location information of cache nodes. For example, the ML model processing engine (142) can receive information that a cache node will be inserted corresponding to a subgraph (P0).

[0045] FIG. 6a illustrates a partitioned graph representation (662) having a cache node (602). Since the cache node will be inserted corresponding to the subgraph (P0), the scheduler (204) may insert the cache node (602) to receive the output of the subgraph (P0). The scheduler (204) may also connect the cache node (602) so that the output of the cache node can be provided to the subgraph (P1). The cache node (602) may represent a memory operation in which the output of the subgraph (P1) is stored in memory and may be retrieved at a later stage during the execution of the model. For example, the memory operation may store the output of the subgraph (P0) in system memory (108) (Fig. 1), local memory of the CPU (104), or local memory of one or more of the ML accelerators.

[0046] FIG. 6b illustrates an exemplary execution data flow graph (650) based on the partitioned graph representation (462) illustrated in FIG. 6a. A scheduler can generate the execution data flow graph (650) by unfolding the iterations of the execution of the partitioned graph representation (662). A portion of the execution data flow graph (650) illustrated in FIG. 6b includes four iterations of iteration 0, iteration 1, iteration 2, and iteration 3. In one iteration, the output of subgraph (P1) is fed as input to the subgraph (P0) of the next iteration. The number of iterations in which the partitioned graph representation is executed can be predetermined. The scheduler (204) can process the partitioned graph representation to determine whether any subgraph should be excluded from execution for each iteration. The scheduler (204) may also consider the requested location and the cache input insertion frequency. For example, in FIG. 6a and FIG. 6b, cache nodes are inserted every odd-numbered iteration. In other words, the model's execution will consume the cached input on every odd iteration.

[0047] As discussed in relation to FIGS. 4a and 4b, the scheduler (204) may determine duplicate nodes or subgraphs and remove them from execution. The scheduler (204) may determine whether a cache node provides an input to a consuming subgraph based on the cache input insertion frequency for each iteration. If a cache node provides an input to a consuming subgraph in an iteration, the cache node is executed to provide an input to the consuming subgraph, and the subgraph that generated the input stored in the cache is excluded from execution. For example, in FIG. 6b, since subgraph (P0) generated an input stored in the cache node in iteration 1, subgraph (P0) is excluded from execution in iteration 1. For a first iteration in which the cache node does not provide an input to the consuming subgraph, the scheduler (204) determines whether the cache node provides an input to the consuming subgraph in the next (second) iteration. If the cache node provides input to the consuming subgraph in the second iteration, the cache node consumes the output from the generating subgraph in the first iteration. For example, in iteration 0, which is an even-numbered iteration in FIG. 6b, the cache node does not provide input to the consuming subgraph, and in the next iteration, iteration 1, the cache node provides input to the consuming subgraph (P1). Thus, in iteration 0, the cache node is executed to consume the output of the subgraph (P0). Additionally, for an iteration in which the cache node does not provide input to the consuming subgraph, the scheduler (204) determines whether the cache node does not provide input to the consuming subgraph in the next iteration, and if not, the cache node is excluded from execution in this iteration. In the execution data flow graph (650) illustrated in FIG. 6b, there is no iteration in which the cache node does not provide input to the consuming subgraph in the next iteration.Accordingly, the cache node is executed in each iteration, but in even iterations, the cache node is executed to store the output of the subgraph (P0), and in odd iterations, the cache node is executed to retrieve the P0 output and provide the P0 output as input to the subgraph (P1).

[0048] In some cases, the scheduler (204) may use a data flow liveness / accessibility analysis algorithm to determine whether a node or subgraph should be marked as non-executable. For example, algorithms such as dead-code deletion, dead-code removal, and dead-code stripping may be used to determine which subgraphs are redundant and can be removed from execution. These algorithms are well known and commonly used in compiler optimization. For example, referring again to FIG. 6b, in iteration 1, the subgraph (P0) is excluded (because, as discussed above, the stored cache node data is provided to the subgraph (P1) instead of the output of P0). In iteration 0, since the output of the subgraph (P1) is provided only to the subgraph (P0) in iteration 1, the subgraph (P1) may also be excluded from being dead. Since P0 is excluded from execution in iteration 1, the execution of the subgraph (P1) in iteration 0 may be redundant and excluded.

[0049] FIG. 7 illustrates a flowchart of an exemplary process (700). The process (700) can be executed by an ML model processing engine (142) (Fig. 1). The process includes a step (702) of determining that a partitioned graph representation of a neural network forms a directed acyclic graph, wherein the partitioned graph representation includes connected subgraphs representing at least one layer of the neural network. At least one example of this process step has been discussed herein in relation to FIGS. 2 through 5b. For example, referring to FIG. 3c, the data flow graph processing engine (202) can determine whether the partitioned graph representation (362) is a DAG. Also, referring to FIG. 5b, the data flow graph processing engine (202) can determine whether the partitioned graph representation (562) is a DAG.

[0050] The process (700) also includes the step (704) of inserting a plurality of cache nodes into a partitioned graph representation, wherein each cache node corresponds to a respective subgraph of a connected subgraph and is configured to cache the output of each subgraph during model execution. At least one example of this process step has been discussed herein in connection with FIGS. 2 through 6b. For example, in FIG. 4b, a cache node (402) is inserted into a partitioned graph representation (462), wherein the cache node corresponds to a subgraph (P1). Similarly, in FIG. 6a, a cache node (602) is inserted into a partitioned graph representation (662), wherein the cache node corresponds to a subgraph (P0).

[0051] The process (700) also includes the step (706) of generating an execution data flow graph comprising multiple iterations of a graph representation partitioned by a computing device, wherein, during one or more iterations during a model inference operation, the execution data flow graph uses input from one or more of the cache nodes and excludes the execution of a subgraph corresponding to one or more cache nodes. At least one example of this process step has been discussed herein in connection with FIGS. 2 through 6b. For example, FIG. 4b illustrates an execution data flow graph (452) over multiple iterations. The execution data flow graph (452) uses input from a cache node (C) at every third iteration. Additionally, the execution data flow graph (452) excludes the execution of a subgraph (P1) at iteration 2, wherein input for the subgraph (P2) is provided from the cache node (C). The execution data flow graph (452) also excludes the execution of a cache node (C) at iteration 0.

[0052] The process (700) further includes a step (708) of compiling an execution data flow graph to generate runtime code for execution on one or more statically compiled machine learning accelerators. At least one example of this process step has been discussed above in relation to FIGS. 2 through 6b. For example, referring to FIGS. 2 and FIGS. 4b, the compiler (206) may extract a compilable representation of each subgraph (P0, P1, P2) and C and statically compile each subgraph individually. Statically compiling the subgraphs individually removes dynamic properties where some nodes are executed and others are not executed during the compilation phase. Dynamic properties are shifted to runtime, where a decision to execute the subgraph is made based on whether the subgraph was marked as non-executable during a specific iteration. As a result, a caching method may be implemented for ML accelerators that do not support dynamic control flow or conditional flow execution.

[0053] In some examples, data cached during one or more iterations may include the output tensor of the subgraph corresponding to the cache node. The output tensor is the result of a neural network computation based on weights and activations received by the subgraph.

[0054] In some examples, for each iteration of multiple iterations, the execution data flow graph excludes the execution subgraph from being fed to the cache node that feeds to another subgraph. For example, referring to FIG. 4b, in iteration 2, the subgraph (P1) is excluded from execution because the subgraph (P1) is fed to the cache node and the cache node is fed again to the subgraph (P2). Since the output of P1 is never used in this iteration, the execution of the subgraph (P1) can be excluded in this iteration.

[0055] In some examples, the execution data flow graph excludes the execution of a subgraph by marking the subgraph so that it is not executed during the runtime for that iteration. In particular, the scheduler (204) may mark the subgraph (P1) in iteration 2 of FIG. 4b so that it is not executed in iteration 2 when the model is executed. The scheduler (204) may generate a schedule of operations for all iterations of the model execution. For each iteration, the schedule may include an indication of which subgraph will not be executed. Alternatively, the schedule may include an indication of which subgraph will be executed. In either case, the compiler or application logic executing the model may refer to the schedule at runtime and determine whether to execute the subgraph during the iteration based on the indication.

[0056] In some examples, the process may include a step of determining the frequency of iterations in which a cache node corresponding to a specific subgraph provides input to a subsequent subgraph. As discussed above in relation to FIG. 4b, the ML model processing engine (142) may receive as input the cache input insertion frequency for every three iterations, starting from iteration 2. The cache input insertion frequency may vary for different implementations. In some cases, the frequency may be periodic, such as every three iterations. In some cases, the cache input insertion frequency input may include multiple frequencies for different sets of iterations, such as every three iterations for the first 20 iterations, and every even iteration for iterations 21 through 40. In some other examples, the cache input insertion frequency may include a list of iterations in which the cache node provides input to a subgraph. For example, the list may include numbers such as 1, 3, 5, 10, ... indicating the number of iterations. Based on the repetition frequency, the ML model processing engine (142) can determine the repetitions in which the cache node provides input to the subgraph.

[0057] In some examples, the process may include a step of determining, based on the iteration frequency, that in a first iteration, a cache node provides a first input to a subsequent subgraph and that a specific subgraph will be excluded during model execution. For example, referring to FIG. 4b, in iteration 2, the cache node (C) provides an input to subgraph (P2), and a specific subgraph—subgraph (P1)—is excluded during model execution. The process may further include a step of determining in a second iteration that a specific subgraph will provide a second input to both the cache node and the subsequent subgraph. For example, referring again to FIG. 4b, the ML model processing engine (142) may determine in iteration 1 that subgraph (P1) will provide an input to the cache node (C) and subgraph (P2), respectively. The process further includes a step of determining in a third iteration that a specific subgraph provides a third input to a subsequent subgraph and that the cache node will be excluded during model execution. For example, the ML model processing engine (142) can determine that in iteration 0, the subgraph (P1) will provide input to the subgraph (P2) and the cache node (C) will not be executed.

[0058] In some examples, the process may include a step of determining that a second partitioned graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a DAG. Based on this determination, the process includes a step of stopping the execution of the second denoising diffusion machine learning model. For example, referring to FIG. 3b, the partitioned graph representation (350) does not form a DAG. As a result, the ML model processing engine (142) may stop the execution of the model.

[0059] The components and processes discussed herein may be implemented on a computing system. In particular, a computing system including a computing device and / or a mobile computing device may be used to implement the technology described herein. For example, one or more processes, electronic design tools, and data may be implemented on or stored thereon on a computing device or a mobile computing device.

[0060] Computing devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Mobile computing devices are intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components, their connections and relationships, and their functions described herein are merely examples and are not intended to be limiting.

[0061] The computing device includes a processor, memory, storage device, a high-speed interface connected to memory and multiple high-speed expansion ports, and a low-speed interface connected to low-speed expansion ports and storage device. Each of the processor, memory, storage device, high-speed interface, high-speed expansion port, and low-speed interface may be interconnected using various buses and mounted on a common motherboard or, if appropriate, in other ways. The processor may process instructions for execution within the computing device, including instructions stored in memory or storage device to display graphic information for a GUI on an external input / output device, such as a display coupled to the high-speed interface. In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and memory types, if appropriate. Additionally, multiple computing devices may be connected, each device providing part of the operation (e.g., as a server bank, a blade server group, or a multiprocessor system). In some embodiments, the processor is a single-threaded processor. In some embodiments, the processor is a multi-threaded processor. In some embodiments, the processor is a quantum computer.

[0062] Memory stores information within a computing device. In some embodiments, memory is a volatile memory unit or units. In some embodiments, memory is a non-volatile memory unit or units. Memory may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0063] A storage device may provide mass storage for a computing device. In some embodiments, the storage device may be or may include an array of devices comprising computer-readable media, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or devices in a storage area network or other configuration. Instructions may be stored in an information carrier. When executed by one or more processing units (e.g., a processor), instructions perform one or more methods as described above. Instructions may also be stored by one or more storage devices, such as a computer or machine-readable media (e.g., memory, a storage device, or memory on a processor). A high-speed interface manages bandwidth-intensive operations of the computing device, while a low-speed interface manages low-bandwidth-intensive operations. The assignment of these functions is merely an example. In some embodiments, a high-speed interface is coupled to a high-speed expansion port capable of accommodating memory, a display (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In some embodiments, a low-speed interface is coupled to a storage device and a low-speed expansion port. A low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be combined with one or more input / output devices, such as a keyboard, pointing device, scanner, or networking device such as a switch or router, for example, via a network adapter.

[0064] The computing device may be implemented in a number of different forms as illustrated in the drawings. For example, it may be implemented multiple times as a standard server or as a group of such servers. Additionally, it may be implemented as a personal computer, such as a laptop computer. It may also be implemented as part of a rack server system. Alternatively, components of the computing device may be combined with other components within a mobile device, such as a mobile computing device. Each of these devices may include one or more of the computing device and the mobile computing device, and the entire system may consist of multiple computing devices communicating with each other.

[0065] A mobile computing device includes input / output devices such as a processor, memory, and display, a communication interface, a transceiver, and other components. To provide additional storage, the mobile computing device may also be provided with a storage device, such as a microdrive or other device. Each processor, memory, display, communication interface, and transceiver is interconnected using various buses, and multiple components may be mounted on a common motherboard or in other appropriate ways.

[0066] The processor can execute instructions, including instructions stored in memory, within the mobile computing device. The processor may be implemented as a chipset of chips comprising multiple separate analog and digital processors. The processor may provide, for example, coordination of other components of the mobile computing device, such as a user interface, applications executed by the mobile computing device, and control of wireless communication by the mobile computing device.

[0067] The processor can communicate with the user through a control interface and a display interface coupled to the display. The display may be, for example, a TFT (Thin-Film Transistor Liquid Crystal Display) display or an OLED (Organic Light-Emitting Diode) display, or other suitable display technology. The display interface may include suitable circuitry for driving the display to present graphics and other information to the user. The control interface may receive commands from the user and convert them to submit to the processor. Additionally, an external interface may be provided for communication with the processor to enable short-range communication with other devices of the mobile computing device. The external interface may provide wired communication in some embodiments or wireless communication in others, for example, and multiple interfaces may be used.

[0068] Memory stores information within a mobile computing device. Memory may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Extended memory may be provided and may be connected to the mobile computing device through an extended interface that may include, for example, a SIMM (Single Inline Memory Module) card interface. Extended memory may provide additional storage space for the mobile computing device or may store applications or other information for the mobile computing device. Specifically, extended memory may include instructions that perform or supplement the processes described herein, and may include security information. Thus, for example, extended memory may be provided as a security module of the mobile computing device and may be programmed with instructions that allow secure use of the mobile computing device. Additionally, security applications may be provided through the SIMM card, along with additional information, for example by placing identification information on the SIMM card in a hack-resistant manner.

[0069] As will be discussed below, memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory). In some embodiments, the instruction is stored in an information carrier, and when executed by one or more processing units (e.g., a processor), the instruction performs one or more methods as described above. The instruction may also be stored by one or more storage devices, such as one or more computers or machine-readable media (e.g., memory, extended memory, or memory on a processor). In some embodiments, the instruction may be received as a signal propagated through, for example, a transceiver or an external interface.

[0070] A mobile computing device may communicate wirelessly through a communication interface, which may include a digital signal processing circuitry in some cases. The communication interface may provide communication under various modes or protocols such as GSM voice calling (Global System for Mobile Communications), SMS (Short Message Service), EMS (Extended Messaging Service) or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Private Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000 or GPRS (General Packet Radio Service), LTE, and 4G / 5G / 6G cellular. Such communication may occur, for example, through a transceiver using radio frequencies. Additionally, short-range communication may occur using, for example, Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module can provide additional navigation and location-related wireless data to a mobile computing device, which can be appropriately used by applications running on the mobile computing device.

[0071] A mobile computing device may also communicate in a way that can be heard by using an audio codec capable of receiving information spoken by a user and converting it into usable digital information. The audio codec can likewise generate sound that can be heard by a user, for example, through a speaker, for example, through a handset of a mobile computing device. Such sound may include sound from a voice phone call, may include recorded sound (e.g., voice messages, music files, etc.), and may include sound generated by an application running on the mobile computing device.

[0072] A mobile computing device may be implemented in a number of different forms as illustrated in the drawings. For example, it may be implemented as a cellular phone. It may also be implemented as part of a smartphone, a personal digital assistant, or other similar mobile device.

[0073] The subject matter of the invention and embodiments of functional operation described herein may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including structures disclosed herein and their structural equivalents, or in a combination of one or more of these. Embodiments of the subject matter of the invention described herein may be implemented as one or more computer programs, namely, as one or more modules of computer program instructions encoded on a tangible non-transient storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded in artificially generated radio signals, for example, machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a receiver device suitable for execution by a data processing device.

[0074] The term “data processing device” refers to data processing hardware and encompasses all types of devices, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be a special-purpose logic circuit, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), or may further include such. Optionally, in addition to the hardware, a device may include code that creates an execution environment for a computer program, for example, processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0075] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed as a standalone program or in any form containing modules, components, subroutines, or other units suitable for use in a computing environment. A program may correspond to a file within a file system, but is not required to do so. A program may be stored in other programs or data, for example, in a part of a file containing one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple coordinated files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program may be deployed to be executed on a single computer or a single site, or on multiple computers distributed across multiple sites and interconnected by a data communication network.

[0076] Process and logic flow may be performed by special-purpose logic circuits, for example, FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers. A computer suitable for executing computer programs may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory or random access memory, or both. Essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by special-purpose logic circuits or integrated into special-purpose logic circuits. Generally, the computer will also include one or more mass storage devices for storing data, for example, magnetic, magneto-optical disks, or optical disks, or will be operably coupled to receive data from them, transmit data to them, or perform both. However, the computer does not need to have such devices. Additionally, the computer can be embedded in other devices, for example, a few examples such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, for example, Universal Serial Bus (USB) flash drives.

[0077] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices including CD-ROM and DVD-ROM disks. — Although this specification contains details of many specific embodiments, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be assigned to specific embodiments of a particular invention. Specific features described in this specification in the context of distinct embodiments may be implemented in combination in a single embodiment. Conversely, in the context of a single embodiment, various features described may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a specific combination and may even be initially claimed as such, one or more features from the claimed combination may, in some cases, be separated from the combination, and the claimed combination may also be directed toward a sub-combination or a variation of the sub-combination.

[0078] Similarly, although operations are illustrated in the drawings and cited in a specific order in the claims, this should not be understood as requiring that such operations be performed in the specific order illustrated or in a sequential order, or that all illustrated operations be performed, in order to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0079] Specific embodiments of the subject have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in different orders and still achieve desirable results. As an example, the process illustrated in the accompanying drawings does not necessarily require the specific order or sequential order illustrated to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

Claim 1 A method for executing a denoising diffusion machine learning model on one or more machine learning accelerators, wherein the denoising diffusion machine learning model is implemented using a neural network, and the method comprises the step of determining by a computing device that a segmented graph representation of the neural network forms a directed acyclic graph—the segmented graph representation comprises connected subgraphs each representing at least one layer of the neural network—; A method comprising: inserting one or more cache nodes into the partitioned graph representation by the computing device—each cache node corresponds to a respective subgraph of the connected subgraph and is configured to cache the output of each subgraph during model execution—; generating an execution data flow graph comprising a plurality of iterations of the partitioned graph representation by the computing device—in one or more iterations during model inference operation, the execution data flow graph uses input from the one or more cache nodes and excludes execution of the subgraph corresponding to the one or more cache nodes—; and compiling each distinct subgraph in the execution data flow graph individually by the computing device to generate runtime code for execution on one or more machine learning accelerators. Claim 2 A method according to claim 1, wherein the output of each subgraph during model execution is an output tensor. Claim 3 A method according to claim 1, wherein for each iteration of the plurality of iterations, the execution data flow graph is excluded from the execution subgraph that feeds to the cache node that feeds to another subgraph. Claim 4 A method according to claim 1, wherein the execution data flow graph excludes the execution of a subgraph by marking the subgraph so that it is not executed during the runtime for the iteration. Claim 5 A method according to claim 1, wherein one or more machine learning accelerators execute the plurality of iterations during a model inference operation based on code compiled individually for each subgraph. Claim 6 A method according to claim 1, further comprising: a step of determining an iteration frequency at which a cache node corresponding to a specific subgraph provides an input to a subsequent subgraph; and a step of determining whether the cache node provides an input to the subsequent subgraph based on the iteration frequency. Claim 7 In paragraph 6, Based on the above iteration frequency: a step of determining, in a first iteration, that the cache node provides a first input to the subsequent subgraph and that the specific subgraph will be excluded during model execution; a step of determining, in a second iteration, that the specific subgraph will provide a second input to the cache node and the subsequent subgraph, respectively; and a step of further comprising, in a third iteration, that the specific subgraph provides a third input to the subsequent subgraph and that the cache node will be excluded during model execution. Claim 8 A method according to claim 1, wherein the segmented graph representation is a first segmented graph representation and the denoising diffusion machine learning model is a first denoising diffusion machine learning model, and the method comprises: determining that a second segmented graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a directed acyclic graph; and stopping the execution of the second denoising diffusion machine learning model. Claim 9 One or more non-transient computer-readable storage media storing instructions that, when executed by one or more processors, cause said one or more processors to perform an operation for executing a denoising diffusion machine learning model on one or more machine learning accelerators, wherein said denoising diffusion machine learning model is implemented using a neural network, and said operation is an operation by a computing device to determine that a segmented graph representation of said neural network forms a directed acyclic graph—said that the segmented graph representation includes connected subgraphs each representing at least one layer of said neural network—; A computer-readable storage medium comprising: an operation of inserting one or more cache nodes into the partitioned graph representation by the computing device—each cache node corresponds to a respective subgraph of the connected subgraph and is configured to cache the output of each subgraph during model execution—; an operation of generating an execution data flow graph comprising a plurality of iterations of the partitioned graph representation by the computing device—in one or more iterations during model inference operation, the execution data flow graph uses input from the one or more cache nodes and excludes execution of the subgraph corresponding to the one or more cache nodes—; and an operation of compiling each distinct subgraph in the execution data flow graph individually to generate runtime code for execution on one or more machine learning accelerators by the computing device. Claim 10 In claim 9, a computer-readable storage medium in which the output of each subgraph during model execution is an output tensor. Claim 11 A computer-readable storage medium, wherein, for each iteration of the plurality of iterations, the execution data flow graph is excluded from the execution subgraph that feeds to a cache node that feeds to another subgraph. Claim 12 In claim 9, the execution data flow graph excludes the execution of a subgraph by marking the subgraph so that it is not executed during the runtime for the iteration, for the iteration, a computer-readable storage medium. Claim 13 In claim 9, the computer-readable storage medium wherein the one or more machine learning accelerators execute the plurality of iterations during a model inference operation based on code individually compiled for each subgraph. Claim 14 A computer-readable storage medium according to claim 9, wherein the operation further comprises: an operation of determining an iteration frequency at which a cache node corresponding to a specific subgraph provides an input to a subsequent subgraph; and an operation of determining whether the cache node provides an input to the subsequent subgraph based on the iteration frequency. Claim 15 A computer-readable storage medium according to claim 14, wherein the operation further comprises, based on the iteration frequency: in a first iteration, the cache node providing a first input to the subsequent subgraph and determining that the specific subgraph will be excluded during model execution; in a second iteration, the specific subgraph determining that the specific subgraph will provide a second input to each of the cache node and the subsequent subgraph; and in a third iteration, the specific subgraph providing a third input to the subsequent subgraph and determining that the cache node will be excluded during model execution. Claim 16 A computer-readable storage medium according to claim 9, wherein the segmented graph representation is a first segmented graph representation and the denoising diffusion machine learning model is a first denoising diffusion machine learning model, and the operation further comprises: an operation of determining that a second segmented graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a directed acyclic graph; and an operation of stopping the execution of the second denoising diffusion machine learning model. Claim 17 A system comprising: one or more processors; and one or more storage devices storing instructions for performing operations to execute a denoising diffusion machine learning model on one or more machine learning accelerators when executed by the one or more processors, wherein the denoising diffusion machine learning model is implemented using a neural network, and the operations are, by a computing device, operations determining that a partitioned graph representation of the neural network forms a directed acyclic graph—the partitioned graph representation includes connected subgraphs each representing at least one layer of the neural network—; A system comprising: an operation by the computing device of inserting one or more cache nodes into the partitioned graph representation—each cache node corresponds to a respective subgraph of the connected subgraph and is configured to cache the output of each subgraph during model execution—; an operation by the computing device of generating an execution data flow graph comprising a plurality of iterations of the partitioned graph representation—in one or more iterations during model inference operation, the execution data flow graph uses input from the one or more cache nodes and excludes execution of the subgraph corresponding to the one or more cache nodes—; and an operation by the computing device of individually compiling each distinct subgraph in the execution data flow graph to generate runtime code for execution on one or more machine learning accelerators. Claim 18 In paragraph 17, a system in which the output of each subgraph during model execution is an output tensor. Claim 19 In paragraph 17, for each iteration of the plurality of iterations, the execution data flow graph is excluded from the execution subgraph that feeds to the cache node that feeds to another subgraph. Claim 20 In paragraph 17, the system excludes the execution of a subgraph by marking the execution data flow graph so that, for a repetition, the subgraph is not executed during the runtime for said repetition. Claim 21 In paragraph 17, the system wherein one or more machine learning accelerators execute the plurality of iterations during a model inference operation based on code compiled individually for each subgraph. Claim 22 A system according to claim 17, wherein the above operation further comprises: an operation in which a cache node corresponding to a specific subgraph determines an iteration frequency for providing input to a subsequent subgraph; and an operation in which, based on the iteration frequency, the cache node determines whether to provide input to the subsequent subgraph. Claim 23 A system according to claim 22, wherein the operation further comprises, based on the iteration frequency: in a first iteration, the cache node providing a first input to the subsequent subgraph and determining that the specific subgraph will be excluded during model execution; in a second iteration, determining that the specific subgraph will provide a second input to each of the cache node and the subsequent subgraph; and in a third iteration, determining that the specific subgraph will provide a third input to the subsequent subgraph and the cache node will be excluded during model execution. Claim 24 A system according to claim 17, wherein the partitioned graph representation is a first partitioned graph representation and the denoising diffusion machine learning model is a first denoising diffusion machine learning model, and the operation includes: an operation of determining that a second partitioned graph representation of a neural network implementing a second denoising diffusion machine learning model does not form a directed acyclic graph; and an operation of stopping the execution of the second denoising diffusion machine learning model.

Images