Node Symmetry in Machine Learning Compiler Optimization

Abstract

A processor-implemented method for compiler optimization using node symmetry includes receiving a representation of an artificial neural network (ANN), the representation including a plurality of nodes connected via a plurality of edges. One or more symmetry sets of the nodes are determined based on one or more of a set of attributes for each node or connectivity of the nodes via the edges. One or more orders or schedules for executing the nodes are generated based on the one or more symmetry sets of the nodes.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS)

[0001] This application claims priority to U.S. patent application Ser. No. 18 / 086,611, filed Dec. 21, 2022, entitled "NODE SYMMETRY IN MACHINE LEARNING COMPILER OPTIMIZATION," the disclosure of which is incorporated herein by reference in its entirety.

[0002] Aspects of the present disclosure generally relate to compilers. [Background technology]

[0003]

[0003] An artificial neural network may comprise an interconnected group of artificial neurons (e.g., neuron models). An artificial neural network may be a computing device or may be represented as a method to be executed by a computing device. Artificial neural networks have numerous applications, including feedforward neural networks, convolutional neural networks (CNNs), transformers, graph neural networks (GNNs), recurrent neural networks (RNNs), etc. Specifically, these neural network architectures are used in various technologies such as image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks.

[0004]

[0004] Given the many useful applications of neural networks, there is an increasing demand for their use on edge devices such as smartphones. However, edge devices have limited computational resources, and generalized models may require more complex networks and more computation. Therefore, the memory footprint and high latency of neural networks make their use difficult, especially for efficient deployment and inference on resource-limited devices.

[0005]

[0005] One approach to reducing latency in neural networks is through machine learning accelerators. Machine learning accelerators include specialized processors designed to accelerate machine learning calculations, such as multiply-accumulate operations in matrix-matrix and matrix-vector operations. To employ a machine learning accelerator, a machine learning compiler provides a mapping from a trained machine learning model to a given machine learning accelerator. The machine learning compiler aims to optimize the mapping from the machine learning model to the accelerator.

[0006]

[0006] Conventional approaches to determining mappings may employ heuristics designed for general input distributions that are not adapted for problem-specific input distributions. Furthermore, the solution space for determining mappings or scheduling for performing the operations of machine learning models may be very large, thereby limiting the application of such optimization techniques. Summary of the Invention

[0007]

[0007] The present disclosure is set forth in independent claims, some aspects of which are set forth in dependent claims.

[0008] In aspects of the present disclosure, a processor-implemented method includes receiving a representation of an artificial neural network (ANN) including a plurality of nodes coupled via a plurality of edges. The method also includes determining one or more symmetric sets of the nodes based on one or more of a set of attributes for each node or connectivity of the nodes via the edges. The method further includes generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of the nodes.

[0009] Another aspect of the present disclosure is directed to an apparatus. The apparatus includes a memory and one or more processors coupled to the memory. The processor(s) are configured to receive a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges. The processor(s) are also configured to determine one or more symmetric sets of the nodes based on one or more of a set of attributes for each node or connectivity of the nodes via the edges. The processor(s) are further configured to generate one or more orders or schedules for executing the nodes based on the one or more symmetric sets of the nodes.

[0010] Another aspect of the present disclosure is directed to an apparatus. The apparatus includes means for receiving a representation of an artificial neural network (ANN) including a plurality of nodes coupled via a plurality of edges. The apparatus also includes means for determining one or more symmetric sets of the nodes based on one or more of a set of attributes for each node or connectivity of the nodes via the edges. The apparatus further includes means for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of the nodes.

[0011] In another aspect of the present disclosure, a non-transitory computer-readable medium having program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for receiving a representation of an artificial neural network (ANN) including a plurality of nodes coupled via a plurality of edges. The program code further includes program code for determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or connectivity of the nodes via the edges. The program code still further includes program code for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0012]

[0012] Additional features and advantages of the present disclosure are described below. Those skilled in the art will appreciate that the present disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended to define the scope of the present disclosure. [Brief explanation of the drawings]

[0013]

[0013] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when read in conjunction with the drawings in which like reference characters identify correspondingly throughout. [Figure 1]

[0014] FIG. 1 illustrates an example implementation of a neural network using a system-on-a-chip (SOC) that includes a general-purpose processor, in accordance with some aspects of the present disclosure. [Figure 2A-C]

[0015] 2A-2C are diagrams illustrating neural networks according to embodiments of the present disclosure. [Figure 2D]

[0016] FIG. 1 illustrates an exemplary deep convolutional network (DCN), according to aspects of the present disclosure. [Figure 3]

[0017] FIG. 1 is a block diagram illustrating an example deep convolutional network (DCN), according to aspects of the present disclosure. [Figure 4]

[0018] FIG. 4 is a block diagram illustrating an example software architecture 400 that may modularize artificial intelligence (AI) functionality. [Figure 5]

[0019] FIG. 5A is a block diagram illustrating an example pipeline for generating an order for executing a neural network model through an accelerator.

[0020] FIG. 5B illustrates an exemplary pipeline for scheduling the execution of operations of a computation graph. [Figure 6]

[0021] FIG. 1 illustrates an exemplary portion of a tiled computation graph, according to aspects of the present disclosure. [Figure 7]

[0022] FIG. 1 illustrates an exemplary portion of a real-world computation graph, according to aspects of the present disclosure. [Figure 8]

[0023] FIG. 10 illustrates an exemplary ordering of nodes based on symmetry, according to aspects of the present disclosure. [Figure 9]

[0024] FIG. 10 is a flow diagram illustrating a process for determining a symmetry set according to an aspect of the present disclosure. [Figure 10]

[0025] FIG. 1 is a flow diagram illustrating a processor-implemented method for generating an order for executing nodes of a computation graph, according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014]

[0026] The detailed description, set forth below with reference to the accompanying drawings, illustrates various configurations and does not represent the only configurations in which the described concepts may be practiced. The detailed description includes specific details intended to provide a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0015]

[0027] Based on the teachings, those skilled in the art will understand that the scope of the present disclosure is intended to encompass any aspect of the present disclosure, whether implemented independently of or in combination with any other aspect of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. Additionally, the scope of the present disclosure is intended to encompass such apparatuses or methods practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the present disclosure described. It will be understood that any aspect of the present disclosure that is disclosed may be embodied by one or more elements of a claim.

[0016]

[0028] The word "exemplary" is used to mean "serving as an example, instance, or illustration." Any aspect described as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0017]

[0029] While particular aspects are described, many variations and permutations of these aspects fall within the scope of the present disclosure. While some benefits and advantages of the preferred aspects are described, the scope of the present disclosure is not limited to particular benefits, applications, or objectives. Rather, aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the figures and the following description of the preferred aspects. The Detailed Description and drawings are not limiting but merely illustrative of the present disclosure, the scope of which is defined by the appended claims and their equivalents.

[0018]

[0030] As mentioned above, neural network architectures are used in a variety of technologies, such as image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks. However, the memory footprint and high latency for neural networks make their use difficult for efficient deployment and inference, especially on resource-limited devices.

[0019]

[0031] One approach to reducing latency in neural networks is through machine learning accelerators. To employ a machine learning accelerator, a machine learning compiler provides a mapping from a trained machine learning model to a given machine learning accelerator. The machine learning compiler aims to optimize the mapping from the machine learning model to the accelerator. However, conventional approaches to determining the mapping may use heuristics that are not suitable for problem-specific input distributions, and the solution space for determining the mapping for a machine learning model may be very large, thus limiting the applicability of such optimization techniques.

[0020]

[0032] One goal in machine learning compilers is to find sequencing and scheduling solutions (e.g., mappings) to improve or optimize key performance indicators (KPIs), such as inferences per second (IPS), memory (e.g., dynamic random access memory (DRAM) read / write bandwidth, memory footprint, or power consumption. For clarity, sequencing may refer to determining the sequence or order for execution of operations. Scheduling may refer to determining which operations to execute on given hardware resources (e.g., processors, such as neural processing units (NPUs), or memory).

[0021]

[0033] One challenge in machine learning compilers is the large solution space. The number of valid sequences (which may also be called "topological orders") for a computation graph may be given by O(N!), where N is the total number of nodes in the computation graph, which is a graphical representation of the topology of operations in a neural network model. A computation graph may be generated after tiling the neural network model. Tiling may refer to separating a neural network layer into multiple artificial neurons. These artificial neurons may then be grouped and represented as kernels or nodes.

[0022]

[0034] A computation graph for such a neural network model may have tens of thousands of nodes. The computation graph may include nodes to represent operations within the neural network model. The nodes may be connected by edges, which may represent dependencies (e.g., precedence constraints) or execution order. Additionally, balancing data locality (e.g., for data reuse) and parallelism, given a computation graph and available hardware resources, is a non-deterministic polynomial-time (NP)-hard problem.

[0023]

[0035] Some conventional solutions may attempt to limit the solution space using a depth-first or breadth-first approach. Depth-first search (DFS) is a process for traversing a graph depthwise using a stack data structure. To generate an order, DFS starts from the root node and explores each branch until it reaches a node with no unvisited neighbors and then backtracks to traverse the next branch in the graph. On the other hand, breadth-first search (BFS) generates an order by starting from the root node and exploring all nodes at the same depth before exploring nodes at the next depth level. Depth-first approaches often aim to optimize locality but can result in hardware underutilization and therefore reduced inferences per second (IPS). Breadth-first approaches often aim to optimize parallelism but can result in infeasible solutions (e.g., topological orders that do not satisfy memory constraints).

[0024]

[0036] On the other hand, some conventional solutions aim to use heuristic approaches to balance data locality and parallelism, but the large search space makes such methods ineffective in determining a solution within a limited time period.

[0025]

[0037] To address these and other challenges, aspects of the present disclosure relate to determining an order of execution based on node symmetry. Nodes in a computation graph may be considered symmetric if they have the same or similar attributes or if they have the same or similar node neighborhoods (e.g., connectivity). Symmetric nodes may be grouped into one or more symmetric sets. In some aspects, the one or more symmetric sets may be determined based on a search space size and a degree of symmetry. An order of nodes within each symmetric set may be determined. Because each symmetric set includes a subset of nodes in the computation graph, the order of nodes within each symmetric set may be considered a partial order. Furthermore, the partial order of nodes within each symmetric set may be determined prior to determining a topological order for the entire computation graph. Therefore, the search space may be significantly reduced from the number of nodes in the computation graph. Furthermore, the topological order and scheduling of nodes may be determined based on one or more of precedence constraints or hardware (e.g., memory) constraints.

[0026]

[0038] Aspects of the present disclosure may therefore advantageously enable an improved balance between data locality and parallelism by using node symmetry to reduce the search space, which may then be explored more deeply to determine a more optimal topological order and schedule.

[0027]

[0039] 1 illustrates an example implementation of a system-on-chip (SOC) 100, which may include a central processing unit (CPU) 102 or multi-core CPU configured to determine an order for executing nodes based on node symmetry. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in memory blocks associated with a neural processing unit (NPU) 108, memory blocks associated with the CPU 102, memory blocks associated with a graphics processing unit (GPU) 104, memory blocks associated with a digital signal processor (DSP) 106, memory blocks 118, or may be distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from memory blocks 118.

[0028]

[0040] SOC 100 may also include a connectivity block 110, which may include a GPU 104, a DSP 106, fifth generation (5G) connectivity, fourth generation long-term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., as well as additional processing blocks adapted to specific functions, such as a multimedia processor 112, which may detect and recognize gestures. In one implementation, NPU 108 is implemented in CPU 102, DSP 106, and / or GPU 104. SOC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, and / or a navigation module 120, which may include a global positioning system.

[0029]

[0041] The SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges. The general-purpose processor 102 may also include code for determining one or more symmetry sets of nodes based on one or more of a set of attributes for each node or the connectivity of the nodes via a plurality of edges. Additionally, the general-purpose processor 102 may include code for generating an order for executing the nodes based on the one or more symmetry sets of nodes.

[0030]

[0042] Deep learning architectures may perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction within each layer, thereby building useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Prior to the advent of deep learning, machine learning approaches to object recognition problems may have relied heavily on human-designed features, possibly in combination with shallow classifiers. A shallow classifier may, for example, be a two-class linear classifier that can compare a weighted sum of feature vector components to a threshold to predict which class an input belongs to. Human-designed features may be templates or kernels tailored to a particular problem domain by an engineer with domain expertise. In contrast, deep learning architectures may learn, but through training, to represent features similar to those a human engineer could design. Furthermore, deep networks may learn to represent and recognize new types of features that humans may not have considered.

[0031]

[0043] Deep learning architectures may learn a hierarchy of features. When presented with visual data, for example, a first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0032]

[0044] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, electric vehicle classification may benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher layers to recognize cars, trucks, and airplanes.

[0033]

[0045] Neural networks may be designed with various connectivity patterns. In feedforward networks, each neuron in a given layer communicates with neurons in a higher layer, and information is passed from lower layers to higher layers. As described above, hierarchical representations may be constructed within successive layers of a feedforward network. Neural networks may also have recurrent or feedback (also called top-down) connections. In recurrent connections, outputs from neurons in a given layer may be communicated to other neurons in the same layer. Recurrent architectures may be useful in recognizing patterns across two or more of the input data chunks delivered to the neural network in sequence. Connections from neurons in a given layer to neurons in a lower layer are called feedback (or top-down) connections. Networks with many feedback connections may be useful when recognizing high-level concepts can help identify specific low-level features of the input.

[0034]

[0046] The connections between layers of a neural network can be fully connected or locally connected. FIG. 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, a neuron in a first layer can communicate its output to every neuron in a second layer, resulting in each neuron in the second layer receiving input from every neuron in the first layer. FIG. 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, a neuron in a first layer can be connected to a limited number of neurons in the second layer. More generally, the locally connected layer of a locally connected neural network 204 can be configured so that each neuron in the layer has the same or similar connectivity pattern but different connection strength values ​​(e.g., 210, 212, 214, and 216). Because higher layer neurons in a given region can receive inputs that are tuned through training to the properties of a limited subset of all inputs to the network, the connectivity patterns of local connections can give rise to spatially distinct receptive fields within the higher layers.

[0035]

[0047] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths (e.g., 208) associated with the inputs for each neuron in the second layer are shared. Convolutional neural networks may be suitable for problems in which the spatial location of the inputs is meaningful.

[0036]

[0048] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN 200 designed to recognize visual features from images 226 input from an image capture device 230, such as an in-car camera. The DCN 200 in this example may be trained to identify traffic signs and numbers printed on traffic signs. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or identifying traffic signals.

[0037]

[0049] DCN 200 may be trained using supervised learning. During training, DCN 200 may be presented with an image, such as speed limit sign image 226, and then a forward pass may be computed to generate output 222. DCN 200 may include a feature extraction section and a classification section. Upon receiving image 226, convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate first set of feature maps 218. As an example, the convolutional kernel for convolutional layer 232 may be a 5×5 kernel that generates 28×28 feature maps. In this example, four different feature maps are generated in first set of feature maps 218, so four different convolutional kernels were applied to image 226 in convolutional layer 232. A convolutional kernel may also be referred to as a filter or convolutional filter.

[0038]

[0050] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0039]

[0051] 2D , the second set of feature maps 220 are convolved to generate a first feature vector 224. The first feature vector 224 is then further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign," "60," and "100." A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 contains one or more features.

[0040]

[0052] In this example, the probability in output 222 for "sign" and "60" is higher than the probability for other outputs 222, such as "30," "40," "50," "70," "80," "90," and "100." Prior to training, the output 222 produced by DCN 200 may be inaccurate. Therefore, an error can be calculated between output 222 and a target output. The target output is the ground truth (e.g., "sign" and "60") of image 226. The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 more closely matches the target output.

[0041]

[0053] To adjust the weights, the learning algorithm can calculate a gradient vector for the weights. The gradient can indicate the amount by which the error would increase or decrease if the weights were adjusted. In the top layer, the gradient can correspond directly to the values ​​of the weights connecting activated neurons in the penultimate layer to neurons in the output layer. In lower layers, the gradient can depend on the values ​​of the weights and the calculated error gradients of the upper layers. The weights can then be adjusted so that the error decreases. This method of adjusting the weights is sometimes called "backpropagation" because it involves a "backward pass" through the neural network.

[0042]

[0054] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the overall system stops decreasing or reaches a target level. After training, the DCN can be presented with new images, and a forward pass through the network can produce output 222, which can be considered the DCN's inference or prediction.

[0043]

[0055] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of a training dataset. DBNs can be obtained by stacking layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions without information about which class each input should be classified into, RBMs are frequently used in unsupervised learning. Using a hybrid unsupervised-supervised paradigm, the lower RBM of a DBN can be trained in an unsupervised manner and function as a feature extractor, while the upper RBM can be trained in a supervised manner (on the joint distribution of inputs from previous layers and the target class) and function as a classifier.

[0044]

[0056] Deep convolutional networks (DCNs) are networks of convolutional networks constructed with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for a large number of examples and are used to modify the network weights using gradient descent.

[0045]

[0057] A DCN may be a feedforward network. Additionally, as described above, connections from neurons in a first layer of a DCN to groups of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be exploited for high-speed processing. The computational burden of a DCN may be much less than that of a similarly sized neural network that includes, for example, recurrent or feedback connections.

[0046]

[0058] The processing in each layer of a convolutional network may be viewed as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input may be viewed as three-dimensional, with two spatial dimensions along the image axes and a third dimension capturing color information. The outputs of the convolutional connections may be viewed as forming feature maps in subsequent layers, with each element of the feature map (e.g., 220) receiving input from a range of neurons in the previous layer (e.g., feature map 218) and from each of multiple channels. Values ​​in the feature map may be further processed using nonlinearities such as rectification, max(0,x), etc. Values ​​from neighboring neurons may also be pooled, which corresponds to downsampling and may provide additional local invariance and dimensionality reduction. Normalization, corresponding to whitening, may also be applied through lateral inhibition between neurons within the feature map.

[0047]

[0059] The performance of deep learning architectures can improve as more labeled data points become available or as computational power increases. Modern deep neural networks are routinely trained using computational resources thousands of times greater than those available to a typical researcher just 15 years ago. New architectures and training paradigms can further improve deep learning performance. Rectified linear units may reduce the training problem known as vanishing gradients. New training techniques may reduce overfitting, thus allowing larger models to achieve better generalization. Encapsulation techniques extract data within a given receptive field, further improving overall performance.

[0048]

[0060] 3 is a block diagram illustrating a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0049]

[0061] The convolutional layer 356 may include one or more convolutional filters that may be applied to input data to generate feature maps. While only two of the convolutional blocks 354A, 354B are shown, the present disclosure is not so limited; instead, any number of convolutional blocks 354A, 354B may be included in the deep convolutional network 350 according to design preference. The normalization layer 358 may normalize the outputs of the convolutional filters. For example, the normalization layer 358 may provide whitening or lateral inhibition. The max-pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0050]

[0062] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 350 may access other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0051]

[0063] The deep convolutional network 350 may also include one or more fully connected layers 362 (FC1 and FC2). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 are weights (not shown) that are updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as the input of a subsequent layer in the deep convolutional network 350 (e.g., 356, 358, 360, 362, 364) to learn a hierarchical feature representation from input data 352 (e.g., image, audio, video, sensor data, and / or other input data) initially provided at the convolution block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, each probability being the probability that the input data contains a feature from the set of features.

[0052]

[0064] 4 is a block diagram illustrating an example software architecture 400 that can modularize artificial intelligence (AI) functions. Using that architecture, applications can be designed that can cause various processing blocks (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) of a system-on-chip (SOC) 420 to support adaptive rounding as disclosed for post-training quantization for AI applications 402, according to aspects of the present disclosure.

[0053]

[0065] The AI ​​application 402 may be configured to invoke functions defined in user space 404, which may, for example, perform scene detection and recognition indicative of the location in which the device is currently operating. The AI ​​application 402 may configure microphones and cameras differently depending, for example, on whether the scene to be recognized is an office, an auditorium, a restaurant, or an outdoor setting such as a lake. The AI ​​application 402 may make requests to compiled program code associated with libraries defined in the AI ​​function application programming interface (API) 406. The requests may ultimately rely on the output of, for example, a deep neural network configured to provide an inference response based on the video and positioning data.

[0054]

[0066] A runtime engine 408, which may be compiled code of a runtime framework, may also be accessible to the AI ​​application 402. The AI ​​application 402 may cause the runtime engine to request inference, for example, at specific time intervals or triggered by an event detected by the application's user interface. When the runtime engine is prompted to provide an inference response, the runtime engine may in turn signal an operating system in the operating system (OS) space, such as the Linux kernel 412 running on the SOC 420. The operating system may then cause successive relaxations of quantization to be performed on the CPU 422, the DSP 424, the GPU 426, the NPU 428, or some combination thereof. The CPU 422 may be accessed directly by the operating system, and other processing blocks may be accessed through drivers, such as drivers 414, 416, or 418 for the DSP 424, the GPU 426, or the NPU 428, respectively. In an illustrative example, the deep neural network may be configured to run on a combination of processing blocks such as CPU 422, DSP 424, and GPU 426, or may run on NPU 428.

[0055]

[0067] An application 402 (e.g., an AI application) may be configured to invoke functions defined in user space 404, which may, for example, perform scene detection and recognition indicative of the location in which the device is currently operating. The application 402 may configure microphones and cameras differently depending, for example, on whether the scene to be recognized is an office, an auditorium, a restaurant, or an outdoor setting such as a lake. The application 402 may make a request to compiled program code associated with a library defined in SceneDetect application programming interface (API) 406 to provide an estimate of the current scene. This request may ultimately rely on the output of, for example, a differential neural network configured to provide a scene estimate based on video and positioning data.

[0056]

[0068] A runtime engine 408, which may be compiled code of a runtime framework, may also be accessible to the application 402. The application 402 may request a scene estimate from the runtime engine, for example, at specific time intervals or triggered by an event detected by the application's user interface. The runtime engine may then signal an operating system 410, such as a Linux kernel 412 running on the SOC 420, when a scene needs to be estimated. The operating system 410 may then cause calculations to be performed on a CPU 422, a DSP 424, a GPU 426, an NPU 428, or some combination thereof. The CPU 422 may be accessed directly by the operating system, and other processing blocks may be accessed through drivers, such as drivers 414-418 for the DSP 424, the GPU 426, or the NPU 428. In an illustrative example, a differential neural network may be configured to run on a combination of processing blocks, such as the CPU 422 and the GPU 426, or may run on the NPU 428.

[0057]

[0069] As described above, aspects of the present disclosure relate to determining one or more of an execution order or schedule based on node symmetry. According to aspects of the present disclosure, symmetric nodes in a computation graph may be used to reduce the search space. This is because real-world computation graphs may have many regularities. That is, many nodes in a computation graph may look similar. For example, many nodes in a computation graph may have the same or similar attributes or neighborhood structure (e.g., connectivity). The tiling process separates neural network layers into multiple nodes. Each node in a given layer may have the same or similar attributes, such as path length, output size, node type, duration, or other attributes. In addition, each node in a layer may have the same neighborhood structure, such as connectivity to tiled nodes from other neural network layers. Therefore, using node symmetry may advantageously result in a significant reduction in the search space. Due to the reduced search space and / or improved search due to the reduced search space and / or learning process, for example, to balance locality and parallelism may be employed. Thus, aspects of the present disclosure may reduce latency and power consumption, and may improve other key performance indicators (KPIs) for machine learning (ML) compilers.

[0058]

[0070] FIG. 5A is a block diagram illustrating an example pipeline 500 for generating an order for executing a neural network model via an accelerator. Referring to FIG. 5A, the pipeline 500 includes an ML framework module 502, an ML compiler 504, and an ML accelerator 506. The ML framework module 502 receives data 508 and uses the data 508 to generate an ML model. The data 508 may include, for example, numerical data (e.g., continuous or discrete data), categorical data (e.g., data representing characteristics), time series data, text, or other data types. In some aspects, the data may be preprocessed to format raw data, for example, to address missing data or detect outliers in the data. The ML framework module 502 processes the data 508 to train an ML model, for example, based on the data type. The ML framework module may employ supervised learning (e.g., regression or classification) or unsupervised learning (e.g., clustering) to train the ML model. The ML model may then be provided to the ML compiler 504. The ML compiler 504, in turn, generates machine code for running the ML model on a given ML accelerator 506. The ML accelerator may include, for example, a neural processing unit (NPU), a digital signal processor (DSP), or a tensor processing unit (TPU).

[0059]

[0071] FIG. 5B illustrates an example pipeline 550 for scheduling the execution of operations of a computation graph. Referring to FIG. 5B, a computation graph 552 is shown. The computation graph 552 includes multiple nodes (e.g., a data-in node, nodes O1-O5, and a data-out node) connected by a set of edges. In some aspects, the computation graph 552 may be, for example, a directed acyclic graph (DAG). A DAG is a finite, directed graph with no directed cycles. That is, each edge is associated with a direction from the start node to the end node indicated by the edge's arrow, and there are no closed loops formed along any path in the graph.

[0060]

[0072] Various constraints 554 may be observed to generate a schedule for executing nodes in the computation graph 552. For example, hardware (HW) resource availability (e.g., types of threads (e.g., accelerator data (movement) threads (ADMTs), accelerator compute threads (ACTs)) within a processor (NPU, DSP, TPU), number of processors, or memory size), as well as precedence constraints (e.g., node dependencies) and other hardware constraints may be used to generate the schedule. A schedule 556 may be generated and may start by executing data within a node via an accelerator data movement thread (ADMT). Thereafter, when hardware is available and memory does not exceed memory constraints (e.g., memory capacity), subsequent nodes in the computation graph 552 may be assigned to accelerator compute processing threads (e.g., ADMTs and ACTs), as well as accelerators. Nodes may be added to the schedule 556 until all nodes in the graph are included in the schedule 556. As discussed above, one goal is to reduce, and in some aspects optimize, the schedule so that the latency for executing the computation graph 552 (e.g., shown as makespan in the schedule) can be reduced.

[0061]

[0073] FIG. 6 illustrates an example portion of a tiled computation graph 600, according to an embodiment of the present disclosure. Tiling may refer to separating a neural network layer into multiple artificial neurons. These artificial neurons are grouped and represented as kernels or nodes. As discussed above, embodiments of the present disclosure may exploit similarities and regularities in computation graph structure. Referring to FIG. 6, the tiled computation graph 600 may highlight similarities between nodes because many of the nodes have similar connectivity. For example, input node 602 (e.g., O A , O B , and O C)Each of the input nodes 602 has a connection from a previous tiled layer (not shown) and has similar connectivity (e.g., connectivity 1) to a set of hidden nodes 604. The hidden nodes 604 likewise have similar connectivity (e.g., connectivity 2) to each other. That is, each of the nodes has a connection to a corresponding input node 602 and an output node 606. Furthermore, because each of the output nodes has a connection from each of a corresponding set of hidden nodes 604 and a connection to a successor node in the next tiled layer (not shown), the output nodes 606 have similar connectivity to each other (e.g., connectivity 3). Because the input nodes 602 have similar connectivity, the input nodes may be considered symmetric.

[0062]

[0074] In some aspects, nodes with the same or similar attributes (e.g., path length, output size, node type, duration, or other attributes) may also be considered symmetric. Additionally, nodes in the same neural network layer may have the same attributes. Furthermore, nodes in the same neural network layer may have the same neighborhood structure. Nodes that are symmetric may be grouped into symmetric sets. A partial order for execution may be determined for each symmetric set. For example, in some aspects, the partial order may be determined using a graph layout process. The graph layout process may generate position information for a graphical representation of the graph. The position information may then be stored and used as a partial order for each symmetric set.

[0063]

[0075] As another example, in some aspects, a partial order may be determined using a topological sort, such as, for example, a depth-first search, a breadth-first search, or the like. Depth-first search (DFS) is a process for traversing a graph depthwise using a stack data structure. To generate an order, DFS starts from the root node and explores each branch until it reaches a node with no unvisited neighbors and then backs off to traverse the next branch in the graph. On the other hand, breadth-first search (BFS) generates an order by starting from the root node and exploring all nodes at the same depth before exploring nodes at the next depth level. The topological sort of nodes within each symmetric set may then be used as a partial order. By ordering nodes based on their symmetric sets rather than the entire computation graph, the search space may be reduced.

[0064]

[0076] FIG. 7 illustrates an exemplary portion of a real-world computation graph 700 according to an embodiment of the present disclosure. As shown in FIG. 7, the computation graph 700 includes multiple nodes 702 connected by edges. For simplicity and ease of explanation, only nodes 702a-702d and 702u-702z are labeled. However, it should be understood that other boxes in FIG. 7 also represent nodes of the computation graph 700. Similarly, only one edge 704 is labeled. However, it should be understood that each arrow between nodes represents an edge. Additionally, for ease of explanation, a number, which may be referred to as the level of the graph, is listed to the left of each row of nodes. The computation graph 700 may be received, for example, as input to a compiler (e.g., 504 in FIG. 5A). In some embodiments, the computation graph 700 may be a directed acyclic graph.

[0065]

[0077] Each of the nodes 702a-702z may represent an operation to be performed, for example, within an artificial neural network (e.g., 350 shown in FIG. 3). Each of the nodes (e.g., 702a-702z) includes a set of attributes. For example, each node may include attributes such as an identification (ID) number, a layer number, a group number, a size, a designation of a hardware device for execution (e.g., ADMT, ACT), or a duration. As described above, nodes with the same or similar attributes may be considered symmetric and may be included in the same symmetric set. For example, each of the input nodes (e.g., level 1 nodes 702a-702d) may have the same size (e.g., A), the same duration (e.g., B), and designate the same hardware device for execution (e.g., ADMT). Because these input nodes (e.g., level 1 nodes 702a-702d) have the same attributes, these input nodes (e.g., level 1 nodes 702a-702d) may be considered symmetric and may be included in the same symmetric set (indicated by the same pattern fill).

[0066]

[0078] In some aspects, nodes with the same or similar neighborhoods (e.g., connectivity) may also be included in the same symmetric set. For example, nodes 702v, 702x, and 702y each have connections (e.g., edges) from a predecessor node at level 2 and a predecessor node at level 3. Based on the similar connectivity, nodes 702v, 702x, and 702y may be included in the same symmetric set. Similarly, nodes 702u, 702w, and 702z have no connections from predecessors and may be included in the same symmetric set. In this manner, nodes in the computation graph 700 may be included in one or more symmetric sets. In some aspects, symmetric sets may also be determined based on considerations of the degree of symmetry or the search space size. An order for executing nodes in each symmetric set may then be determined. While the symmetric sets shown in FIG. 7 include nodes within the same level, this is merely for ease of explanation and is not a limitation. It should be understood that nodes within different levels (e.g., 1-4) may instead be included in the same symmetric set.

[0067]

[0079] FIG. 8 is a diagram 800 illustrating an example ordering of nodes based on symmetry, according to an embodiment of the present disclosure. Referring to FIG. 8, a simple computation graph 802a includes six nodes for which an order is to be determined. As indicated in chart 804, the search space before applying node symmetry includes 16 possible solutions (e.g., an order, which may also be referred to as a topological order). As described above, the search space may be reduced based on node symmetry. However, in some embodiments, the degree of symmetry used to determine the symmetry set may be adjusted to further reduce the search space. In computation graph 802b, stricter symmetry (e.g., stronger symmetry) is applied. For example, in computation graph 802b, node 1 and node 3 have the same connectivity. That is, both nodes have a connection to a single successor node (e.g., node 1 is only connected to node 4, and node 3 is only connected to node 5). On the other hand, node 2 is connected to two successor nodes (e.g., node 4 and node 5). Because stronger symmetry applies (e.g., nodes may be considered symmetric if they have the same connectivity as opposed to similar connectivity), node 2 is not included in the same symmetry set as node 1 and node 3. Ordering the nodes in each symmetry set, node 1 and node 3, may be performed before node 2. Thus, the search space may be reduced to five possible orderings (shown in the Stronger Symmetry column of chart 804) taking into account precedence constraints (e.g., dependencies between nodes).

[0068]

[0080] In computation graph 802c, a looser symmetry (e.g., weaker symmetry) is applied. For example, in computation graph 802b, node 1 and node 3 have the same connectivity. That is, both nodes have a connection to a single successor node (e.g., node 1 is only connected to node 4, and node 3 is only connected to node 5). Like node 1, node 2 is connected to node 4, and like node 3, node 2 is connected to successor node 5, so node 2 has connectivity similar to that of nodes 1 and 3. Therefore, because a looser symmetry (e.g., weaker symmetry) is applied, nodes 1 through 3 may be included in the same symmetry set. Accordingly, the search space may be further reduced to two possible orders (e.g., shown in the Weaker Symmetry column of chart 804) taking precedence constraints into account. After the search space is reduced, an order may be selected from the possible orders using, for example, a graph layout process, a DFS process, a BFS process, or a topological sort.

[0069]

[0081] Thus, the degree of symmetry can act as a constraint for reducing the search space. For example, this can be advantageously applied to reduce the search space based on computational power (e.g., processing speed, memory size), such that weaker symmetries can be adopted when operating on architectures with lower computational power, and stronger symmetries can be adopted when operating on architectures with higher computational power.

[0070]

[0082] 9 is a flow diagram illustrating a process for determining a symmetry set according to an aspect of the present disclosure. Referring to FIG. 9, at block 902, process 900 may optionally define the degree of symmetry or search space reduction to be applied. For example, by using a larger number of node features, a stricter (stronger) symmetry may be defined. Conversely, by using a fewer number of node features, a looser (weaker) symmetry may be defined.

[0071]

[0083] At block 904, the process 900 begins with the initial node feature c (0)(n) to each node n, where 0 represents the iteration in the node embedding definition in Equation 1. For example, features may include node attributes such as the type of node operation (DMA, NPU, DSP, etc.), path length (shortest / longest) from the source node, output tensor size, duration, or other attributes.

[0072]

[0084] At block 906, the process 900 generates a node embedding for each node based on the node features. A node embedding can be thought of as a representation of the node as a vector. For example, a node embedding can be expressed as:

[0073]

number

[0074]

[0085] Using the exemplary node embedding defined in Equation 1, a stricter (stronger) symmetry can be defined by using an injective function for H for node embeddings based on a larger number of iterations K and a larger number of node features. On the other hand, a looser (weaker) symmetry can be defined by using an injective function for H for node embeddings based on a larger number of iterations K and a smaller number of node features.

[0075]

[0086] At block 908, process 900 may optionally iteratively update each of the node embeddings based on other node embeddings (e.g., embeddings of neighboring nodes).

[0076]

[0087] At block 910, the process 900 assigns nodes to one or more symmetric sets based on the node embedding. For example, (k+1) Nodes with , can be assigned to the same symmetric set.

[0077]

[0088] Furthermore, according to aspects of the present disclosure, after determining the symmetric set, a partial order for executing each of the nodes in the symmetric set may be determined. In some aspects, the partial order may be determined using, for example, a graph layout process, a tie-breaking process, a sequencing process, or a combination thereof. Using this partial order, the search space may be reduced so that an order and / or schedule for executing the nodes of the computation graph may be further determined. The partial order refers to the order within the symmetric set, whereas the order refers to the complete order of the graph.

[0078]

[0089] FIG. 10 is a flow diagram illustrating a processor-implemented method 1000 for generating an order for executing nodes of a computation graph, according to an aspect of the present disclosure. As shown in FIG. 10, at block 1002, the processor-implemented method 1000 receives a representation of an artificial neural network (ANN) including multiple nodes connected via multiple edges. For example, as described above with reference to FIG. 7, the computation graph 700 may be received, for example, as input to a compiler (e.g., 504). Using a computation graph may advantageously enable further reductions in power consumption and latency due to sparsity in the representation. In some aspects, the representation of the ANN may include a matrix or another representation of the ANN.

[0079]

[0090] Each of the nodes 702a-702z may represent an operation to be performed, for example, within an artificial neural network (e.g., 350 shown in FIG. 3). Each of the nodes (e.g., 702a-702z) includes a set of attributes. For example, each node may include attributes such as an identification (ID) number, a layer number, a group number, a size, instructions for a hardware device to execute on (e.g., DMA, NPU, TPU), or a duration.

[0080]

[0091] At block 1004, the processor-implemented method 1000 determines one or more symmetric sets of nodes based on one or more of a set of attributes for each node or the connectivity of the nodes through multiple edges. For example, as described above with reference to FIG. 7, nodes with the same or similar attributes may be considered symmetric and included in the same symmetric set. Each of the input nodes (e.g., level 1 nodes 702a-702d) has the same size (e.g., A), the same duration (e.g., B), and indicates the same hardware device for execution (e.g., ADMT). Because these input nodes (e.g., level 1 nodes 702a-702d) have the same attributes, these input nodes (e.g., level 1 nodes 702a-702d) may be considered symmetric and may be included in the same symmetric set (indicated by the same pattern fill). In some aspects, nodes with the same or similar neighborhoods (e.g., connectivity) may also be included in the same symmetric set. For example, nodes 702v, 702x, and 702y each have connections (e.g., edges) from a predecessor node at level 2 and a predecessor node at level 3. Based on similar connectivity, nodes 702v, 702x, and 702y may be included in the same symmetric set. Similarly, nodes 702u, 702w, and 702z have no connections from predecessor nodes and may be included in the same symmetric set.

[0081]

[0092] At block 1006, the processor-implemented method 1000 generates one or more orders or schedules for executing the nodes based on one or more symmetric sets of nodes. For example, as shown in FIG. 8, one or more topological orders may be determined based on the symmetric sets. For example, if stricter symmetry is applied, the search space may be reduced, and five possible orders (e.g., as shown in chart 804) are determined taking into account precedence constraints (e.g., dependencies between nodes). In some aspects, the orders may be schedules for executing the nodes of the computation graph. For example, as shown in FIG. 5B, a schedule 556 for executing the nodes of the computation graph 552 is generated based on various constraints 554. For example, hardware resource availability (e.g., processor type (NPU or TPU), number of processors, or memory size), as well as precedence constraints (e.g., node dependencies) and other hardware constraints may be used to generate the schedule.

[0082] Exemplary Embodiments

[0093] Aspect 1: A processor-implemented method comprising: receiving a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges; determining one or more symmetric sets of the nodes based on one or more of a set of attributes for each node or the connectivity of the nodes via the plurality of edges; and generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of the nodes.

[0083]

[0094] Aspect 2: The processor-implemented method of aspect 1, further comprising: assigning at least one attribute to each node; determining a node embedding for each node based on an aggregation of the attributes for one or more of the predecessor nodes and / or successor nodes; and assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding.

[0084]

[0095] Aspect 3: The processor-implemented method of aspect 1 or 2, further comprising determining one or more symmetric sets of nodes based on a search space size and a degree of symmetry.

[0085]

[0096] Aspect 4: The processor-implemented method of any one of aspects 1-3, further comprising iteratively updating the node embedding for each node based on other node embeddings.

[0086]

[0097] Aspect 5: The processor-implemented method of any one of aspects 1 to 4, wherein one or more of the orders or schedules in each symmetric set are determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

[0087]

[0098] Aspect 6: A processor-implemented method of any one of aspects 1 to 5, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution time, and wherein the processor-implemented method further includes assigning nodes having one or more of the same path length, same node type, same output tensor size, or same node execution time to one of the one or more symmetric sets of nodes.

[0088]

[0099] Aspect 7: The processor-implemented method of any one of aspects 1 to 6, wherein the representation comprises a computation graph.

[0089]

[0100] Aspect 8: The processor-implemented method of any one of aspects 1 to 7, wherein the representation includes a computation graph, and the method further includes generating one or more of an order or schedule for executing nodes of the computation graph via the processing device.

[0090]

[0101] Aspect 9: An apparatus comprising: a memory; and at least one processor coupled to the memory, wherein the at least one processor is configured to receive a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges; determine one or more symmetric sets of the nodes based on one or more of a set of attributes for each node or a connectivity of the nodes via the plurality of edges; and generate one or more orders or schedules for executing the nodes based on the one or more symmetric sets of the nodes.

[0091]

[0102] Aspect 10: The apparatus of aspect 9, wherein at least one processor is further configured to assign at least one attribute to each node, determine a node embedding for each node based on an aggregation of the attributes for one or more of the predecessor nodes and / or successor nodes, and assign the node to one of one or more symmetric sets of nodes based on the determined node embedding.

[0092]

[0103] Aspect 11: The apparatus of aspect 9 or 10, wherein the at least one processor is further configured to determine one or more symmetric sets of nodes based on a search space size and a degree of symmetry.

[0093]

[0104] Aspect 12: The apparatus of any one of aspects 9 to 11, wherein the at least one processor is further configured to iteratively update the node embedding for each node based on other node embeddings.

[0094]

[0105] Aspect 13: The apparatus of any one of aspects 9 to 12, wherein one or more of the orders or schedules in each symmetric set are determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

[0095]

[0106] Aspect 14: The apparatus of any one of aspects 9 to 13, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution time, and wherein the at least one processor is further configured to assign nodes having one or more of the same path length, same node type, same output tensor size, or same node execution time to one of the one or more symmetric sets of nodes.

[0096]

[0107] Aspect 15: The apparatus of any one of aspects 9 to 14, wherein the representation comprises a computation graph.

[0097]

[0108] Aspect 16: The apparatus of any one of aspects 9 to 15, wherein the representation includes a computation graph, and wherein the at least one processor is further configured to generate one or more of an order or schedule for executing nodes of the computation graph via the processing device.

[0098]

[0109] Aspect 17: An apparatus, comprising: means for receiving a representation of an artificial neural network (ANN) including a plurality of nodes coupled via a plurality of edges; means for determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or a connectivity of the nodes via a plurality of edges; and means for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0099]

[0110] Aspect 18: The apparatus of aspect 17, further comprising: means for assigning at least one attribute to each node; means for determining a node embedding for each node based on an aggregation of the attributes for one or more of the predecessor nodes and / or successor nodes; and means for assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding.

[0100]

[0111] Aspect 19: The apparatus of aspect 17 or 18, further comprising means for determining one or more symmetric sets of nodes based on a search space size and a degree of symmetry.

[0101]

[0112] Aspect 20: The apparatus of any one of aspects 17 to 19, further comprising means for iteratively updating the node embedding for each node based on other node embeddings.

[0102]

[0113] Aspect 21: The apparatus of any one of aspects 17 to 20, wherein one or more of the orders or schedules in each symmetric set are determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

[0103]

[0114] Aspect 22: The apparatus of any one of aspects 17 to 21, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution time, and further comprising means for assigning nodes having the same path length, same node type, same output tensor size, or same node execution time to one of one or more symmetric sets of nodes.

[0104]

[0115] Aspect 23: The apparatus of any one of aspects 17 to 22, wherein the representation comprises a computation graph.

[0105]

[0116] Aspect 24: The apparatus of any one of aspects 17 to 23, wherein the representation includes a computation graph, and the apparatus further comprises means for generating one or more of an order or schedule for executing nodes of the computation graph via the processing device.

[0106]

[0117] Aspect 25: A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and including: program code for receiving a representation of an artificial neural network (ANN) including a plurality of nodes coupled via a plurality of edges; program code for determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or a connectivity of the nodes via a plurality of edges; and program code for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0107]

[0118] Aspect 26: The non-transitory computer-readable medium of aspect 25, further including program code for assigning at least one attribute to each node, program code for determining a node embedding for each node based on an aggregation of the attributes for one or more of the predecessor nodes and / or successor nodes, and program code for assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding.

[0108]

[0119] Aspect 27: The non-transitory computer-readable medium of aspect 25 or 26, wherein the program code further includes program code for determining one or more symmetric sets of nodes based on a search space size and a degree of symmetry.

[0109]

[0120] Aspect 28: The non-transitory computer-readable medium of any one of aspects 25 to 27, wherein the program code further includes program code for iteratively updating the node embedding for each node based on other node embeddings.

[0110]

[0121] Aspect 29: A non-transitory computer-readable medium according to any one of aspects 25 to 28, wherein one or more of the orders or schedules in each symmetric set are determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

[0111]

[0122] Aspect 30: A non-transitory computer-readable medium as described in any one of aspects 25 to 29, further comprising program code for assigning nodes having one or more of the same path length, node type, output tensor size, or node execution time to one of one or more symmetric sets of nodes, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution time.

[0112]

[0123] In one aspect, the receiving means, determining means, and / or generating means may be GPU 104, a program memory associated with GPU 104, fully connected layer 362, NPU 428, and / or routing connection processing unit 216 configured to perform the recited functions. In another configuration, the above-mentioned means may be any module or any device configured to perform the recited functions by the above-mentioned means.

[0113]

[0124] The various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s), including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in figures, those operations may have corresponding counterpart means-plus-function components that are similarly numbered.

[0114]

[0125] As used, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., searching a table, database, or another data structure), ascertaining, and the like. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Further, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0115]

[0126] When used, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc.

[0116]

[0127] The various example logic blocks, modules, and circuits described in connection with this disclosure may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the described functions. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0117]

[0128] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in any form of storage medium known in the art. Some examples of storage media that may be used include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, etc. A software module may comprise a single instruction, or multiple instructions, and may be distributed across several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor.

[0118]

[0129] The disclosed methods include one or more steps or actions for achieving the described method. The steps and / or actions of those methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order of specific steps and / or actions and / or the use of those steps and / or actions may be modified without departing from the scope of the claims.

[0119]

[0130] The described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an exemplary hardware configuration may comprise a processing system within a device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnected buses and bridges, depending on the particular application of the processing system and overall design constraints. The bus may link various circuits together, including processors, machine-readable media, and bus interfaces. The bus interface may be used to, among other things, connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In certain aspects, a user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and therefore will not be described further.

[0120]

[0131] The processor may be responsible for managing the bus and general processing, including executing software stored on a machine-readable medium. The processor may be implemented using one or more general-purpose and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry capable of executing software. Software shall be broadly construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The machine-readable medium may include, by way of example, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disk, optical disk, hard drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may comprise packaging materials.

[0121]

[0132] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art will readily appreciate, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all of which may be accessed by the processor through a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as is the case with cache and / or general-purpose register files. While various components discussed may be described as having particular locations, such as local components, these components may also be configured in various ways, such as certain components configured as part of a distributed computing system.

[0122]

[0133] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable medium, all linked together with other support circuitry through an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron model and model of the neural system. As another alternative, the processing system may be implemented using an application-specific integrated circuit (ASIC) having the processor, bus interface, user interface, support circuitry, and at least a portion of the machine-readable medium integrated into a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, or any other suitable circuitry or combination of circuitry capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the described functionality of the processing system depending on the particular application and the overall design constraints imposed on the entire system.

[0123]

[0134] The machine-readable medium may comprise several software modules. The software modules include instructions that, when executed by a processor, cause the processing system to perform various functions. The software modules may include a transmitting module and a receiving module. Each software module may reside in a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a trigger event occurs. During execution of a software module, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring hereinafter to functionality of a software module, it will be understood that such functionality is implemented by the processor upon executing instructions from that software module. Furthermore, it will be understood that aspects of the present disclosure result in improved functionality of a processor, computer, machine, or other system implementing such aspects.

[0124]

[0135] If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Thus, in some aspects computer-readable medium may comprise non-transitory computer-readable medium (e.g., tangible media). In addition, in other aspects computer-readable media may comprise transitory computer-readable media (eg, a signal). Combinations of the above should also be included within the scope of computer-readable media.

[0125]

[0136] Accordingly, certain aspects may comprise a computer program product for performing the presented operations. For example, such a computer program product may comprise a computer-readable medium having stored thereon (and / or encoded thereon) instructions executable by one or more processors to perform the described operations. In certain aspects, the computer program product may include packaging materials.

[0126]

[0137] Furthermore, it should be understood that modules and / or other suitable means for performing the described methods and techniques may be downloaded and / or otherwise obtained by a user terminal and / or base station, where applicable. For example, such devices may be coupled to a server to facilitate the transfer of means for performing the described methods. Alternatively, the various described methods may be provided via a storage means such that the user terminal and / or base station can obtain the various methods upon coupling or providing the storage means (e.g., RAM, ROM, physical storage medium such as a compact disc (CD) or floppy disk, etc.) to the device. Furthermore, any other suitable technique for providing the described methods and techniques to a device may be utilized.

[0127]

[0138] It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. 1. A processor-implemented method comprising: receiving a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges; determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or connectivity of the nodes through the plurality of edges; generating one or more of an order or schedule for executing the nodes based on one or more symmetric sets of the nodes; 11. A processor-implemented method comprising:

2. assigning at least one attribute to each node; determining a node embedding for each node based on an aggregation of attributes for one or more of the predecessor nodes and / or successor nodes; assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding; The processor-implemented method of claim 1 further comprising:

3. The processor-implemented method of claim 2 , further comprising determining one or more symmetric sets of the nodes based on a search space size and a degree of symmetry.

4. The processor-implemented method of claim 2 , further comprising iteratively updating the node embedding for each node based on other node embeddings.

5. 2. The processor-implemented method of claim 1, wherein the one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

6. 2. The processor-implemented method of claim 1 , wherein the set of attributes includes one or more of a path length, a node type, an output tensor size, or a node execution time, and wherein the processor-implemented method further includes assigning the nodes having the same path length, the same node type, the same output tensor size, or the same node execution time to one of the one or more symmetric sets of nodes.

7. The processor-implemented method of claim 1 , wherein the representation comprises a computation graph.

8. 2. The processor-implemented method of claim 1, wherein the representation comprises a computation graph, the processor-implemented method further comprising generating the one or more of the order or the schedule for executing the nodes of the computation graph via a processing device.

9. 1. An apparatus comprising: Memory and at least one processor coupled to the memory, wherein the at least one processor: receiving a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges; determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or connectivity of the nodes through the plurality of edges; generating one or more of an order or schedule for executing the nodes based on one or more symmetric sets of the nodes; The apparatus is configured to:

10. the at least one processor: assigning at least one attribute to each node; determining a node embedding for each node based on an aggregation of attributes for one or more of the predecessor nodes and / or successor nodes; assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding; 10. The apparatus of claim 9, further configured to:

11. The apparatus of claim 10 , wherein the at least one processor is further configured to determine one or more symmetric sets of the nodes based on a search space size and a degree of symmetry.

12. The apparatus of claim 10 , wherein the at least one processor is further configured to iteratively update the node embedding for each node based on other node embeddings.

13. The apparatus of claim 9 , wherein the one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

14. 10. The apparatus of claim 9, wherein the set of attributes includes one or more of a path length, a node type, an output tensor size, or a node execution time, and wherein the at least one processor is further configured to assign the nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution time to one of the one or more symmetric sets of nodes.

15. The apparatus of claim 9 , wherein the representation comprises a computation graph.

16. 10. The apparatus of claim 9, wherein the representation includes a computation graph, and the at least one processor is further configured to generate the one or more of the order or the schedule for executing the nodes of the computation graph via a processing device.

17. 1. An apparatus comprising: means for receiving a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges; means for determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or connectivity of the nodes through the plurality of edges; means for generating one or more of an order or schedule for executing the nodes based on one or more symmetric sets of the nodes; An apparatus comprising:

18. means for assigning at least one attribute to each node; means for determining a node embedding for each node based on an aggregation of attributes for one or more of the predecessor nodes and / or successor nodes; means for assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding; 20. The apparatus of claim 17, further comprising:

19. The apparatus of claim 18 , further comprising: means for determining one or more symmetric sets of the nodes based on a search space size and a degree of symmetry.

20. 20. The apparatus of claim 18, further comprising: means for iteratively updating the node embedding for each node based on other node embeddings.

21. The apparatus of claim 17 , wherein the one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

22. 20. The apparatus of claim 17, wherein the set of attributes comprises one or more of a path length, a node type, an output tensor size, or a node execution time duration, and further comprising means for assigning the nodes having the same path length, the same node type, the same output tensor size, or the same node execution time duration to one of the one or more symmetric sets of nodes.

23. The apparatus of claim 17 , wherein the representation comprises a computation graph.

24. 20. The apparatus of claim 17, wherein the representation includes a computation graph, and wherein the apparatus further comprises means for generating the one or more of the order or the schedule for executing the nodes of the computation graph via a processing device.

25. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor; and program code for receiving a representation of an artificial neural network (ANN) including a plurality of nodes connected via a plurality of edges; program code for determining one or more symmetric sets of nodes based on one or more of a set of attributes for each node or connectivity of the nodes through the plurality of edges; program code for generating one or more of an order or schedule for executing the nodes based on one or more symmetric sets of the nodes; 1. A non-transitory computer-readable medium comprising:

26. The program code: program code for assigning at least one attribute to each node; program code for determining a node embedding for each node based on an aggregation of attributes for one or more of the predecessor nodes and / or successor nodes; program code for assigning the node to one of one or more symmetric sets of nodes based on the determined node embedding; 26. The non-transitory computer-readable medium of claim 25, further comprising:

27. 27. The non-transitory computer-readable medium of claim 26, wherein the program code further comprises program code for determining one or more symmetric sets of the nodes based on a search space size and a degree of symmetry.

28. 27. The non-transitory computer-readable medium of claim 26, wherein the program code further comprises program code for iteratively updating the node embeddings for each node based on other node embeddings.

29. 26. The non-transitory computer-readable medium of claim 25, wherein the one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

30. 26. The non-transitory computer-readable medium of claim 25, further comprising program code for: the set of attributes comprising one or more of a path length, a node type, an output tensor size, or a node execution time; and assigning the nodes having the same path length, the same node type, the same output tensor size, or the same node execution time to one of the one or more symmetric sets of nodes.

Images