Node symmetry in machine learning compiler optimization

Abstract

A processor-implemented method for compiler optimization using node symmetry includes receiving a representation of an artificial neural network (ANN), the representation including a plurality of nodes coupled via a plurality of edges. One or more symmetric sets of nodes are determined based on one or more of the set of attributes for each node or connectivity of the nodes via the edges. One or more of an order or a schedule for executing the nodes is generated based on the one or more symmetric sets of nodes.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 18 / 086,611, filed on December 21, 2022, entitled “NODE SYMMETRY IN MACHINELEARNING COMPILER OPTIMIZATION,” the disclosure of which is expressly incorporated herein by reference in its entirety.

[0003] public domain

[0004] Aspects of the present disclosure generally relate to compilers. Background Art

[0005] An artificial neural network may include a group of interconnected artificial neurons (e.g., a neuron model). An artificial neural network may be a computing device or represented as a method to be executed by a computing device. Artificial neural networks (including feedforward neural networks, convolutional neural networks (CNNs), transformers, graph neural networks (GNNs), recurrent neural networks (RNNs), etc.) have numerous applications. Specifically, these neural network architectures are used in various technologies such as image recognition, speech recognition, acoustic scene classification, keyword detection, autonomous driving, and other classification tasks.

[0006] Given the many useful applications of neural networks, there is a growing demand for using them on edge devices, such as smartphones. However, edge devices have limited computational resources, and generalized models can utilize more complex networks and more computation. Consequently, the memory footprint and high latency of neural networks make their use challenging, especially for efficient deployment and inference on resource-constrained devices.

[0007] One way to reduce latency in neural networks is through machine learning accelerators. Machine learning accelerators include specialized processors designed to accelerate machine learning calculations, such as multiply-accumulate operations in matrix-matrix and matrix-vector operations. To employ machine learning accelerators, machine learning compilers provide a mapping from trained machine learning models to a given machine learning accelerator. Machine learning compilers are designed to optimize the mapping from machine learning models to accelerators.

[0008] Conventional approaches for determining this mapping employ heuristics designed for general input distributions that are not tailored to the problem-specific input distribution. Furthermore, the solution space for determining the mapping or schedule of operations used to execute a machine learning model can be very large, limiting the applicability of such optimization techniques.

[0009] Overview

[0010] The present disclosure is set out in the independent claims. Some aspects of the present disclosure are described in the dependent claims.

[0011] In various aspects of the present disclosure, a processor-implemented method includes receiving a representation of an artificial neural network (ANN) comprising a plurality of nodes coupled via a plurality of edges. The method further includes determining one or more symmetric sets of nodes based on one or more of a set of attributes of each node or the connectivity of the nodes via the edges. The method further includes generating one or more of an order or a schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0012] Other aspects of the present disclosure relate to a device. The device has a memory and one or more processors coupled to the memory. The processor(s) are configured to receive a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges. The processor(s) are further configured to determine one or more symmetric node sets based on one or more of a set of attributes of each node or the connectivity of the nodes via the edges. The processor(s) are further configured to generate one or more of an order or schedule for executing the nodes based on the one or more symmetric node sets.

[0013] Other aspects of the present disclosure relate to an apparatus. The apparatus includes means for receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges. The apparatus also includes means for determining one or more symmetric node sets based on one or more of a set of attributes of each node or the connectivity of the nodes via the edges. The apparatus further includes means for generating one or more of an order or a schedule for executing the nodes based on the one or more symmetric node sets.

[0014] In other aspects of the present disclosure, a non-transient computer-readable medium having program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for receiving a representation of an artificial neural network (ANN), the representation including a plurality of nodes coupled via a plurality of edges. The program code further includes program code for determining one or more symmetric node sets based on a set of attributes of each node or one or more of the connectivity of these nodes via these edges. The program code further includes program code for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric node sets.

[0015] Additional features and advantages of the present disclosure are described below. It will be appreciated by those skilled in the art that the present disclosure can be readily used as a basis for modifying or designing other structures for carrying out the same purposes as the present disclosure. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features which are believed to be characteristic of the present disclosure, both in terms of its organization and method of operation, together with further objects and advantages, will be better understood when the following description is considered in conjunction with the accompanying drawings. However, it is to be clearly understood that each of the figures is provided for illustration and description purposes only and is not intended as a definition of limitations of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features, nature and advantages of the present disclosure will become more apparent when the detailed description set forth below is read in conjunction with the accompanying drawings, in which like reference numerals designate correspondingly throughout.

[0018] Figure 1 An example implementation of a neural network using a system on a chip (SOC) including a general-purpose processor in accordance with certain aspects of the present disclosure is illustrated.

[0019] Figure 2A 、 2B 2C are diagrams illustrating neural networks according to aspects of the present disclosure.

[0020] Figure 2D is a diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure.

[0021] Figure 3 is a block diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure.

[0022] Figure 4 is a block diagram illustrating an exemplary software architecture 400 that may modularize artificial intelligence (AI) functionality.

[0023] Figure 5A is a block diagram illustrating an example pipeline for generating an order for executing a neural network model via an accelerator.

[0024] Figure 5B is a diagram illustrating an example pipeline for scheduling execution of operations of a computation graph.

[0025] Figure 6 is a diagram illustrating an example portion of a tiled computation graph in accordance with aspects of the present disclosure.

[0026] Figure 7 is a diagram illustrating an example portion of a real-world computation graph in accordance with aspects of the present disclosure.

[0027] Figure 8is a diagram illustrating an example ordering of nodes based on symmetry in accordance with aspects of the present disclosure.

[0028] Figure 9 is a flow chart illustrating a process for determining a symmetry set in accordance with aspects of the present disclosure.

[0029] Figure 10 is a flowchart illustrating a processor-implemented method for generating an order for executing nodes of a computation graph according to aspects of the present disclosure.

[0030] Detailed description

[0031] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the described concepts may be practiced. This detailed description includes specific details in order to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0032] Based on this teaching, it will be appreciated by those skilled in the art that the scope of the present disclosure is intended to cover any aspect of the present disclosure, regardless of whether it is implemented independently or in combination with any other aspect of the present disclosure. For example, any number of aspects set forth can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods that are practiced using other structures, functionality, or structure and functionality that are supplementary to or different from the various aspects of the present disclosure set forth. It should be understood that any aspect of the present disclosure disclosed can be implemented by one or more elements of the claims.

[0033] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0034] Although specific aspects have been described, numerous variations and permutations of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects have been mentioned, the scope of this disclosure is not intended to be limited to specific benefits, uses, or objectives. On the contrary, various aspects of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated as examples in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings merely illustrate the disclosure and do not limit it, the scope of which is defined by the appended claims and their equivalents.

[0035] As described, neural network architectures are used in a variety of technologies such as image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks. However, the memory footprint and high latency of neural networks make their use challenging, especially for efficient deployment and inference on resource-constrained devices.

[0036] One approach to reducing latency in neural networks is through machine learning accelerators. To employ machine learning accelerators, machine learning compilers provide a mapping from a trained machine learning model to a given machine learning accelerator. Machine learning compilers aim to optimize the mapping from machine learning models to accelerators. However, conventional approaches to determining this mapping may use heuristics that are inappropriate for the problem-specific input distribution, and the solution for determining this mapping for a machine learning model can be very large, limiting the application of such optimization techniques.

[0037] One goal of a machine learning compiler is to find sequencing and scheduling solutions (e.g., mappings) to improve or optimize key performance indicators (KPIs), such as inferences per second (IPS), memory (e.g., dynamic random access memory (DRAM)) read / write bandwidth, memory footprint, or power consumption, for example. For clarity, sequencing may refer to determining a sequence or order for execution of operations. Scheduling may refer to determining the operations to be executed on a given hardware resource (e.g., a processor (such as a neural processing unit (NPU)) or memory).

[0038] One challenge in machine learning compilers is the vast solution space. The number of valid sequences (also known as "topological orders") of a computation graph can be given by O(N!), where N is the total number of nodes in the computation graph, and where the computation graph is a graphical representation of the topology of operations in a neural network model. The computation graph can be generated after tiling the neural network model. Tiling can refer to separating a neural network layer into multiple artificial neurons. These artificial neurons can then be grouped and represented as kernels or nodes.

[0039] The computation graph for such a neural network model can have tens of thousands of nodes. The computation graph can include nodes representing operations in the neural network model. These nodes can be connected by edges that can represent dependencies (e.g., precedence constraints) or execution order. Additionally, balancing data locality (e.g., for data reuse) and parallelism, given the computation graph and available hardware resources, is a non-deterministic polynomial time (NP) hard problem.

[0040] Some conventional solutions may try to constrain the solution space using a depth-first or breadth-first approach. Depth-first search (DFS) is a process for traversing a graph in the depth direction using a stack data structure. DFS starts from the root node and explores each branch until it reaches a node with no unvisited neighbor nodes, and then backtracks to traverse the next branch in the graph to generate an order. On the other hand, breadth-first search (BFS) starts from the root node and explores all nodes at the same depth before exploring nodes at the next depth level to generate an order. Depth-first approaches typically aim to optimize locality, but may result in insufficient hardware utilization, thereby reducing inferences per second (IPS). Breadth-first approaches typically aim to optimize parallelism, but may result in infeasible solutions (e.g., a topological order that does not satisfy memory constraints).

[0041] On the other hand, some conventional solutions aim to use heuristics to try to balance data locality and parallelism. However, the huge search space makes such methods ineffective in determining a solution within a limited time period.

[0042] To address these and other challenges, various aspects of the present disclosure relate to determining an execution order based on node symmetry. If the nodes in a computation graph have the same or similar properties, or if the nodes in a computation graph have the same or similar node neighborhoods (e.g., connectivity), then the nodes can be considered symmetric. Symmetric nodes can be grouped in one or more symmetric sets. In some aspects, one or more symmetric sets can be determined based on the search space size and the degree of symmetry. The order of the nodes in each symmetric set can be determined. The order of the nodes in each symmetry can be considered a partial order because each symmetric set includes a subset of the nodes of the computation graph. In addition, the partial order of the nodes in each symmetric set can be determined before determining the topological order of the entire computation graph. In this way, the search space can be greatly reduced from the number of nodes in the computation graph. In addition, the topological order and scheduling of the nodes can be determined based on one or more of a precedence constraint or a hardware (e.g., memory) constraint.

[0043] Accordingly, aspects of the present disclosure can advantageously achieve an improved balance between data locality and parallelism by reducing the search space using node symmetry. In this way, the reduced search space can be searched more deeply to determine a more optimal topological order and schedule.

[0044] Figure 1An example implementation of a system on a chip (SOC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to determine an order for executing nodes based on node symmetry. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), latency, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at CPU 102 may be loaded from a program memory associated with CPU 102 or may be loaded from memory block 118.

[0045] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one implementation, the NPU 108 is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120 (which may include a global positioning system).

[0046] The SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving a representation of an artificial neural network (ANN), the representation including a plurality of nodes coupled via a plurality of edges. The general-purpose processor 102 may also include code for determining one or more symmetric node sets based on a set of attributes of each node or one or more of the connectivity of the nodes via the plurality of edges. Additionally, the general-purpose processor 102 may include code for generating an order for executing the nodes based on the one or more symmetric node sets.

[0047] Deep learning architectures perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction at each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before the advent of deep learning, machine learning approaches to object recognition problems might rely heavily on human-engineered features, perhaps combined with shallow classifiers. A shallow classifier might be a two-class linear classifier, for example, where the weighted sum of the feature vector components is compared to a threshold to predict which class the input belongs to. Human-engineered features might be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, deep learning architectures learn to represent features similar to those that a human engineer might design, but they do so through training. Furthermore, deep networks can learn to represent and recognize new types of features that humans might not have considered.

[0048] Deep learning architectures can learn hierarchies of features. For example, if a first layer is presented with visual data, it can learn to recognize relatively simple features (such as edges) in the input stream. In another example, if a first layer is presented with auditory data, it can learn to recognize spectral power in specific frequencies. A second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0049] Deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motor vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0050] Neural networks can be designed to have various connectivity patterns. In a feedforward network, information is passed from lower layers to higher layers, with each neuron in a given layer communicating to neurons in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have reflow or feedback (also known as top-down) connections. In a reflow connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The reflow architecture can help identify patterns that span more than one chunk of input data delivered sequentially to the neural network. The connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. A network with many feedback connections can be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input.

[0051] The connections between layers of a neural network can be fully connected or partially connected. Figure 2A Illustrated is an example of a fully connected neural network 202. In the fully connected neural network 202, a neuron in a first layer may communicate its output to every neuron in a second layer, such that every neuron in the second layer will receive input from every neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in a layer will have the same or similar connectivity pattern, but their connection strengths may have different values (e.g., 210, 212, 214, and 216). The locally connected connectivity pattern may produce spatially distinct receptive fields in higher layers because higher layer neurons in a given area may receive inputs that are tuned through training to properties of a limited portion of the total input to the network.

[0052] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. The convolutional neural network 206 can be configured such that the connection strengths associated with the inputs to each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful.

[0053] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as an onboard camera) is illustrated. The DCN 200 of the current example can be trained to identify traffic signs and the numbers provided on traffic signs. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or identifying traffic lights.

[0054] DCN 200 can be trained using supervised learning. During training, an image (such as image 226 of a speed limit sign) can be presented to DCN 200, and a "forward pass" can then be calculated to produce output 222. DCN 200 can include a feature extraction section and a classification section. Upon receiving image 226, convolution layer 232 can apply a convolution kernel (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolution kernel of convolution layer 232 can be a 5x5 kernel that generates a 28x28 feature map. In this example, since four different feature maps are generated in first set of feature maps 218, four different convolution kernels are applied to image 226 at convolution layer 232. Convolution kernels can also be referred to as filters or convolution filters.

[0055] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (such as 14x14) is smaller than the size of the first set of feature maps 218 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0056] exist Figure 2D In the example of , the second set of feature maps 220 is convolved to generate a first feature vector 224. In addition, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226 (such as "sign," "60," and "100"). A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 includes one or more features.

[0057] In this example, the probabilities for "logo" and "60" in output 222 are higher than the probabilities for other features of output 222 (such as "30," "40," "50," "70," "80," "90," and "100"). Before training, output 222 generated by DCN 200 is likely incorrect. Thus, the error between output 222 and the target output can be calculated. The target output is the true value of image 226 (e.g., "logo" and "60"). The weights of DCN 200 can then be adjusted to more closely align output 222 of DCN 200 with the target output.

[0058] To adjust the weights, the learning algorithm can calculate a gradient vector for the weights. The gradient can indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient can directly correspond to the value of the weight connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient can depend on the value of the weights and the calculated error gradient of the higher layers. The weights can then be adjusted to reduce the error. This way of adjusting weights can be called "backpropagation" because it involves a backward pass in the neural network.

[0059] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method may be called stochastic gradient descent. Stochastic gradient descent may be repeated until the error rate achievable by the entire system has stopped decreasing or until the error rate has reached a target level. After learning, a new image may be presented to the DCN and a forward pass through the network may produce output 222, which may be considered an inference or prediction of the DCN.

[0060] Deep Belief Network (DBN) is a probabilistic model comprising multiple layers of hidden nodes. DBN can be used to extract hierarchical representations of training data sets. DBN can be obtained by stacking multiple layers of restricted Boltzmann machines (RBM). RBM is a type of artificial neural network that can learn probability distributions on input sets. Since RBM can learn probability distributions without information about which class each input should be classified into, RBM is often used in unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of the DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of the input and target class from the previous layer) and can be used as a classifier.

[0061] A deep convolutional network (DCN) is a network of convolutional networks with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the network weights using gradient descent.

[0062] The DCN can be a feedforward network. In addition, as described above, the connections from neurons in the first layer of the DCN to the neuron groups in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of the DCN can be used to perform fast processing. The computational burden of the DCN can be much smaller than, for example, the computational burden of a similarly sized neural network that includes recurrent or feedback connections.

[0063] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on this input can be thought of as three-dimensional, with two spatial dimensions along the axes of the image and a third dimension that captures color information. The output of the convolutional connection can be thought of as forming a feature map in the subsequent layer, each element in which receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values in the feature map can be further processed with nonlinearities (such as rectification, max(0,x)). The values from adjacent neurons can be further pooled (which corresponds to downsampling) and can provide additional local invariance and dimensionality reduction. Normalization can also be applied by lateral inhibition between neurons in the feature map, which corresponds to whitening.

[0064] The performance of deep learning architectures improves as more labeled data points become available or as computing power increases. Modern deep neural networks are routinely trained using thousands of times more computing resources than were available to typical researchers just fifteen years ago. New architectures and training paradigms can further boost deep learning performance. Rectified linear units can alleviate the training problem known as vanishing gradients. New training techniques can reduce overfitting and thus enable larger models to achieve better generalization. Encapsulation techniques can abstract the data within a given receptive field and further improve overall performance.

[0065] Figure 3 3 is a block diagram illustrating a deep convolutional network 350. The deep convolutional network 350 may include multiple layers of different types based on connectivity and weight sharing. Figure 3 As shown in FIG, the deep convolutional network 350 includes convolution blocks 354A and 354B. Each of the convolution blocks 354A and 354B may be configured with a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0066] The convolution layer 356 may include one or more convolution filters that can be applied to the input data to generate a feature map. Although only two convolution blocks 354A and 354B are shown, the present disclosure is not limited thereto, and instead any number of convolution blocks 354A and 354B may be included in the deep convolutional network 350 according to design preferences. The normalization layer 358 may normalize the output of the convolution filter. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximum pooling layer 360 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0067] For example, the parallel filter banks of the deep convolutional network can be offloaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter banks can be offloaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 350 can access other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and the navigation module 120 dedicated to sensors and navigation, respectively.

[0068] The deep convolutional network 350 may also include one or more fully connected layers 362 (FC1 and FC2). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 are weights (not shown) to be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) can be used as input to a subsequent layer (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) supplied from the first convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is a probability that the input data includes a feature from a feature set.

[0069] Figure 4 4 is a block diagram illustrating an exemplary software architecture 400 that can modularize artificial intelligence (AI) functionality. According to aspects of the present disclosure, by using this architecture, an application can be designed that enables various processing blocks (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) of a system on a chip (SOC) 420 to support adaptive rounding for post-training quantization of an AI application 402 as disclosed.

[0070] The AI application 402 may be configured to call functions defined in the user space 404 that, for example, may provide detection and recognition of scenes indicating the current operating location of the device. For example, the AI application 402 may configure the microphone and camera differently depending on whether the identified scene is an office, a lecture hall, a restaurant, or an outdoor environment such as a lake. The AI application 402 may make a request to compiled program code associated with a library defined in the AI function application programming interface (API) 406. The request may ultimately rely on the output of a deep neural network configured to provide an inferred response based on, for example, video and positioning data.

[0071] Runtime engine 408 (which may be compiled code of a runtime framework) may further be accessible to AI application 402. For example, AI application 402 may cause the runtime engine to request inference at a specific time interval or triggered by an event detected by the application's user interface. When the runtime engine is caused to provide an inference response, the runtime engine may in turn send a signal to the operating system (OS) space (such as Linux kernel 412) running on SOC 420. The operating system may in turn cause continuous quantization relaxation to be executed on CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. CPU 422 may be directly accessible to the operating system, while other processing blocks may be accessed through drivers (such as drivers 414, 416, or 418 for DSP 424, GPU 426, or NPU 428, respectively). In an illustrative example, a deep neural network may be configured to run on a combination of processing blocks (such as CPU 422, DSP 424, and GPU 426), or may run on NPU 428.

[0072] Application 402 (e.g., an AI application) can be configured to call functions defined in user space 404 that, for example, can provide detection and recognition of a scene indicating the current operating location of the device. For example, application 402 can configure the microphone and camera differently depending on whether the identified scene is an office, a lecture hall, a restaurant, or an outdoor environment such as a lake. Application 402 can make a request to compiled program code associated with a library defined in a scene detection application programming interface (API) 406 to provide an estimate of the current scene. The request can ultimately rely on the output of a differential neural network configured to provide a scene estimate based on, for example, video and positioning data.

[0073] The runtime engine 408 (which may be compiled code of a runtime framework) may further be accessible to the application 402. For example, the application 402 may cause the runtime engine to request scene estimation at a specific time interval or triggered by an event detected by the application's user interface. Upon causing the runtime engine to estimate the scene, the runtime engine may in turn send a signal to the operating system 410 (such as the Linux kernel 412) running on the SOC 420. The operating system 410 may in turn cause the calculation to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. The CPU 422 may be directly accessible by the operating system, while the other processing blocks may be accessed through drivers (such as drivers 414-418 for the DSP 424, GPU 426, or NPU 428, respectively). In the illustrative example, the differential neural network may be configured to run on a combination of processing blocks (such as the CPU 422 and GPU 426), or may run on the NPU 428.

[0074] As described, various aspects of the present disclosure relate to determining one or more of an order or schedule of execution based on node symmetry. According to various aspects of the present disclosure, symmetric nodes in a computation graph can be used to reduce the search space. This is because real-world computation graphs may have many regularities. That is, many nodes in a computation graph may look similar. For example, many nodes in a computation graph may have the same or similar properties or neighborhood structures (e.g., connectivity). The tiling process separates neural network layers into multiple nodes. For example, each node in a given layer may have the same or similar properties, such as path length, output size, node type, duration, or other properties. In addition, each node in the layer may have the same neighborhood structure, such as connectivity to nodes tiled from other neural network layers. In this way, using node symmetry can beneficially provide a significant reduction in the search space. Because the search space is reduced, for example, an improved search due to the reduced search space and / or learning process can be used to balance locality and parallelism. Therefore, various aspects of the present disclosure can reduce latency and power consumption, and can improve other key performance indicators (KPIs) of machine learning (ML) compilers.

[0075] Figure 5A is a block diagram illustrating an example pipeline 500 for generating an order for executing a neural network model via an accelerator. Figure 5A The pipeline 500 includes an ML framework module 502, an ML compiler 504, and an ML accelerator 506. The ML framework module 502 receives data 508 and uses the data 508 to generate an ML model. For example, the data 508 may include numerical data (e.g., continuous data or discrete data), categorical data (e.g., data representing a characteristic), time series data, text, or other data types. In some aspects, the data may be pre-processed, for example, to format the raw data, address missing data, or detect outliers in the data. The ML framework module 502 processes the data 508 to train an ML model, for example, based on the data type. The ML framework module may employ supervised learning (e.g., regression or classification) or unsupervised learning (e.g., clustering) to train the ML model. The ML model may then be provided to the ML compiler 504. The ML compiler 504, in turn, generates machine code for operating the ML model on a given ML accelerator 506. For example, the ML accelerator may include a neural processing unit (NPU), a digital signal processor (DSP), or a tensor processing unit (TPU).

[0076] Figure 5B is a diagram illustrating an example pipeline 550 for scheduling execution of operations of a computation graph. Figure 5B, a computation graph 552 is shown. The computation graph 552 includes a plurality of nodes (e.g., a data input node, nodes O1-O5, and a data output node) connected by a set of edges. In some aspects, for example, the computation graph 552 can be a directed acyclic graph (DAG). A DAG is a finite directed graph with no direct cycles. That is, each edge is associated with a direction from a starting node to an ending node indicated by the arrow of the edge, and no path along the graph forms a closed loop.

[0077] In order to generate a schedule for executing the nodes of the computation graph 552, various constraints 554 may be observed. For example, the availability of hardware (HW) resources (e.g., the type of thread in the processor (NPU, DSP, TPU) (e.g., accelerator data (movement) thread (AMDT), accelerator compute thread (ACT)), the number of processors, or the memory size) and precedence constraints (e.g., node dependencies) and other hardware constraints may be used to generate the schedule. A schedule 556 may be generated and may begin with executing the data in the node via the accelerator data movement thread (ADMT). Thereafter, when the hardware is available and the memory does not exceed the memory constraints (e.g., memory capacity), subsequent nodes in the computation graph 552 may be assigned to accelerator compute processing threads (e.g., ADMT and ACT) and accelerators. Nodes may be added to the schedule 556 until all nodes in the graph are included in the schedule 556. As described, one goal is to reduce and in some aspects optimize the schedule so that the wait time (e.g., considered as a makespan in the schedule) for executing the computation graph 552 can be reduced.

[0078] Figure 6 is a diagram illustrating an example portion of a tiled computation graph 600 according to aspects of the present disclosure. Tiling may refer to separating a neural network layer into multiple artificial neurons. These artificial neurons are grouped and represented as kernels or nodes. As described, aspects of the present disclosure may exploit similarities and regularities in the computation graph structure. Figure 6 , the tiled computation graph 600 can highlight similarities between nodes because many of these nodes have similar connectivity. For example, input node 602 (e.g., A , O B and O C) has a connection from a previous tiled layer (not shown) and has similar connectivity to the set of intermediate nodes 604 (e.g., connectivity 1). The intermediate nodes 604 also have similar connectivity to each other (e.g., connectivity 2). That is, each node has a connection to the corresponding input node 602 and output node 606. In addition, the output nodes 606 have similar connectivity to each other (e.g., connectivity 3) because each of these output nodes has a connection from each intermediate node 604 in the corresponding set of intermediate nodes 604 and a connection to the successor node in the next tiled layer (not shown). Because the input nodes 602 have similar connectivity, these input nodes can be considered symmetrical.

[0079] In some aspects, nodes with the same or similar properties (e.g., path length, output size, node type, duration, or other properties) can also be considered symmetric. Additionally, nodes of the same neural network layer can have the same properties. Furthermore, nodes of the same neural network layer can have the same neighborhood structure. Symmetric nodes can be grouped in symmetric sets. A partial order for execution can be determined for each symmetric set. For example, in some aspects, a graph layout process can be used to determine the partial order. The graph layout process can generate positioning information for a schematic representation of the graph. This positioning information can then be sorted and used as a partial order for each symmetric set.

[0080] As another example, in some aspects, a topological sort (e.g., such as depth-first search, breadth-first search, etc.) can be used to determine a partial order. Depth-first search (DFS) is a process for traversing a graph in a depth direction using a stack data structure. DFS starts from the root node and explores each branch until it reaches a node with no unvisited neighbor nodes, and then backtracks to the next branch in the graph to generate an order. On the other hand, breadth-first search (BFS) starts from the root node and explores all nodes at the same depth before exploring the nodes at the next depth level to generate an order. The topological sort of the nodes in each symmetric set can then be used as a partial order. By sorting the nodes based on the symmetric set of the nodes rather than the entire computational graph, the search space can be reduced.

[0081] Figure 7 is a diagram illustrating an example portion of a real-world computation graph 700 according to aspects of the present disclosure. Figure 7 As shown in FIG, the computation graph 700 includes a plurality of nodes 702 connected by edges. For the sake of brevity and ease of illustration, only nodes 702a-d and 702u-z are labeled. However, it should be understood that Figure 7The other boxes in also represent nodes of the computation graph 700. Similarly, only one edge 704 is labeled. However, it should be understood that each arrow between nodes represents an edge. Additionally, for ease of explanation, a number is included to the left of each row of nodes, which may be referred to as the level of the graph. For example, the computation graph 700 may be used as a compiler (e.g., Figure 5A In some aspects, the computation graph 700 can be a directed acyclic graph.

[0082] Each of the nodes 702a-z may represent an operation to be performed (e.g., Figure 3 350) shown in ). Each of the nodes (e.g., 702a-z) includes a set of attributes. For example, each node may include attributes such as an identification (ID) number, a layer number, a group number, a size, an indication of a hardware device for execution (e.g., ADMT, ACT), or a duration. As described, nodes with the same or similar attributes may be considered to be symmetrical and included in the same symmetry set. For example, each of the input nodes (e.g., level 1 nodes 702a-d) has the same size (e.g., A), the same duration (e.g., B), and indicates the same hardware device for execution (e.g., ADMT). Because these input nodes (e.g., level 1 nodes 702a-d) have the same attributes, these input nodes (e.g., level 1 nodes 702a-d) may be considered to be symmetrical and may be included in the same symmetry set (indicated by the same pattern fill).

[0083] In some aspects, nodes with the same or similar neighborhood (e.g., connectivity) may also be included in the same symmetric set. For example, nodes 702v, 702x, and 702y each have connections (e.g., edges) from a level 2 predecessor node and a level 3 predecessor node. Based on similar connectivity, nodes 702v, 702x, and 702y may be included in the same symmetric set. Similarly, nodes 702u, 702w, and 702z do not have connections from predecessor nodes and may be included in the same symmetric set. In this way, the nodes of computation graph 700 may be included in one or more symmetric sets. In some aspects, symmetric sets may also be determined based on the degree of symmetry or search space size considerations. Furthermore, the order in which the nodes in each symmetric set are executed may be determined. Although Figure 7 The symmetric sets shown in FIG include nodes at the same level, but this is for ease of explanation only and is not limiting. On the contrary, it should be understood that nodes at different levels (eg, 1-4) may be included in the same symmetric set.

[0084] Figure 8 is a diagram 800 illustrating an example ordering of nodes based on symmetry according to aspects of the present disclosure. Figure 8 , simple calculation graph 802a includes six nodes whose order is to be determined. As indicated in chart 804, the search space before applying node symmetry includes sixteen possible solutions (e.g., order, which may also be referred to as topological order). As described, the search space can be reduced based on node symmetry. However, in some aspects, the degree of symmetry used to determine the symmetric set can be adjusted to further reduce the search space. In calculation graph 802b, stricter symmetry (e.g., stronger symmetry) is applied. For example, in calculation graph 802b, nodes 1 and 3 have the same connectivity. That is, both nodes have a connection to a single successor node (e.g., node 1 is only connected to node 4 and node 3 is only connected to node 5). On the other hand, node 2 is connected to two successor nodes (e.g., node 4 and node 5). Because stronger symmetry is applied (e.g., if the nodes have the same connectivity rather than similar connectivity, the nodes can be considered symmetrical), node 2 is not included in the same symmetric set as nodes 1 and 3. The nodes in each symmetric set are sorted, and nodes 1 and 3 can be executed before node 2. Accordingly, the search space can be reduced to five possible orders (shown in the stronger symmetry column of diagram 804) in view of precedence constraints (eg, dependencies between nodes).

[0085] In computation graph 802c, a looser symmetry (e.g., weaker symmetry) is applied. For example, in computation graph 802b, nodes 1 and 3 have the same connectivity. That is, both nodes have a connection to a single successor node (e.g., node 1 is only connected to node 4 and node 3 is only connected to node 5). Node 2 has connectivity similar to that of nodes 1 and 3 because, like node 1, node 2 is connected to node 4, and, like node 3, node 2 is connected to successor node 5. Thus, because a looser symmetry (e.g., weaker symmetry) is applied, nodes 1-3 can be included in the same symmetric set. Accordingly, the search space can be further reduced to two possible orders (e.g., as shown in the weaker symmetry column of graph 804) in view of the precedence constraint. After reducing the search space, an order can be selected from the possible orders using, for example, a graph layout process, a DFS process, a BFS process, or a topological sort.

[0086] Accordingly, the degree of symmetry can be used as a control to reduce the search space. For example, this can be beneficially applied to reduce the search space based on computing power (e.g., processing speed, memory size), such that if operating on an architecture with less computing power, weaker symmetry can be employed, while if operating on an architecture with more computing power, stronger symmetry can be employed.

[0087] Figure 9is a flow chart illustrating a process for determining a symmetry set according to aspects of the present disclosure. Figure 9 At block 902, process 900 may optionally define a degree of symmetry or search space reduction to be applied. For example, a stricter (stronger) symmetry may be defined by using more node features. Conversely, a looser (weaker) symmetry may be defined by using fewer node features.

[0088] At block 904, process 900 assigns an initial node feature c to each node n. (0) (n), where 0 represents the iteration in the node embedding definition of Equation 1. For example, the features may include node attributes such as node operation type (DMA, NPU, DSP, etc.), path length (shortest / longest) from the source node, output tensor size, duration, or other attributes.

[0089] At block 906, process 900 generates a node embedding for each node based on the node features. The node embedding can be viewed as a representation of the node as a vector. For example, the node embedding can be given by:

[0090] c (k) (n)=H({c (k-1) (n), {c (k-1) (u)} u∈N(n)}), k=1,…,K (1)

[0091] where c (K) (n) represents the node embedding at iteration k, H is a function that maps the aggregate embedding of a set of nodes to a value, and N(n) represents the neighbors of node n and may include the predecessor nodes and / or successor nodes of node n. In some aspects, H may include an injective function. An injective function maps an element of a given set to a different element of another set. From the previous node embedding (e.g., c (k-1) (u)) can be combined with its own embedding (e.g., c (k-1) (n)) aggregation to update each node embedding, where the variable u represents the index of the node in the neighborhood of node n.

[0092] Using the example node embedding defined in Equation 1, for node embeddings based on more node features, stricter (stronger) symmetries can be defined by using injective functions with larger iteration numbers K and H. On the other hand, for node embeddings based on fewer node features, looser (weaker) symmetries can be defined by using injective functions with larger iteration numbers K and H.

[0093] At block 908 , process 900 may optionally iteratively update each of these node embeddings based on other node embeddings (eg, embeddings of neighboring nodes).

[0094] At block 910, process 900 assigns each node to one or more symmetric sets based on node embedding. For example, nodes with the same embedding value c (k+1) Nodes can be assigned to the same symmetric set.

[0095] Furthermore, according to aspects of the present disclosure, after determining the symmetric set, a partial order can be determined for executing each node in the symmetric set. In some aspects, the partial order can be determined using, for example, a graph layout process, a tiebreaker process, a sequencing process, or a combination thereof. Using this partial order, one or both of the order or scheduling of nodes for executing the computation graph can be further determined because the search space is reduced. A partial order refers to an order within the symmetric set, while an order refers to a complete order of the graph.

[0096] Figure 10 1 is a flowchart illustrating a method 1000 implemented by a processor for generating an order for executing nodes of a computation graph according to aspects of the present disclosure. Figure 10 As shown in FIG, at block 1002, the processor-implemented method 1000 receives a representation of an artificial neural network (ANN) comprising a plurality of nodes coupled via a plurality of edges. Figure 7 As described, computation graph 700 may be received as input to, for example, a compiler (e.g., 504). Due to the sparsity within the representation, using the computation graph may advantageously achieve further reductions in power consumption and latency. In some aspects, the representation of the ANN may include a matrix or another representation of the ANN.

[0097] Each of the nodes 702a-z may represent an operation to be performed (e.g., Figure 3 350) shown in ). Each of the nodes (e.g., 702a-z) includes a set of attributes. For example, each node may include attributes such as an identification (ID) number, a layer number, a group number, a size, an indication of a hardware device for execution (e.g., DMA, NPU, TPU), or a duration.

[0098] At block 1004, the processor-implemented method 1000 determines one or more symmetric node sets based on one or more attributes in the attribute set of each node or the connectivity of the nodes via the plurality of edges. Figure 7Described, the node with identical or similar attribute can be considered as symmetric, and is included in identical symmetric set.Each in all input nodes (for example, the node 702a-d of level 1) has identical size (for example, A), identical duration (for example, B), and indicates identical hardware device (for example, ADMT) for execution.Because these input nodes (for example, the node 702a-d of level 1) have identical attribute, so these input nodes (for example, the node 702a-d of level 1) can be considered as symmetric, and can be included in identical symmetric set (by identical pattern filling indication).In some respects, the node with identical or similar neighborhood (for example, connectivity) also can be included in identical symmetric set.For example, node 702v, 702x and 702y each have the connection (for example, edge) from level 2 predecessor node and level 3 predecessor node.Based on similar connectivity, node 702v, 702x and 702y can be included in identical symmetric set. Similarly, nodes 702u, 702w, and 702z have no connections from predecessor nodes and may be included in the same symmetric set.

[0099] At block 1006, the processor-implemented method 1000 generates one or more of an order or schedule for executing nodes based on the one or more symmetric node sets. Figure 8 As shown in , one or more topological orders can be determined based on the symmetry set. For example, when stricter symmetry is applied, the search space can be reduced and five possible orders (e.g., as shown in diagram 804) are determined in view of the precedence constraints (e.g., dependencies between nodes). In some aspects, the order can be a schedule for executing the nodes of the computation graph. For example, as Figure 5B As shown in FIG, a schedule 556 for executing nodes of a computation graph 552 is generated based on various constraints 554. For example, the schedule can be generated using the availability of hardware resources (e.g., processor type (NPU or TPU), number of processors, or memory size) as well as precedence constraints (e.g., node dependencies) and other hardware constraints.

[0100] Example aspects

[0101] Aspect 1: A processor-implemented method comprising: receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; determining one or more symmetric sets of nodes based on one or more of a set of attributes of each node or the connectivity of the nodes via the plurality of edges; and generating one or more of an order or a schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0102] Aspect 2: The processor-implemented method of Aspect 1 further includes: assigning at least one attribute to each node; determining a node embedding for each node based on an aggregation of attributes of one or more of the predecessor nodes and / or successor nodes; and assigning the nodes to one of the one or more symmetric node sets based on the determined node embeddings.

[0103] Aspect 3: The processor-implemented method of aspect 1 or 2 further comprises: determining the one or more symmetric node sets based on the search space size and the degree of symmetry.

[0104] Aspect 4: The processor-implemented method of any of the preceding aspects, further comprising: iteratively updating the node embedding of each node based on the other node embeddings.

[0105] Aspect 5: The processor-implemented method of any of the preceding aspects, wherein the one or more of the order or schedule in each symmetry set is determined based on at least one of a graph layout process, a tiebreaker process, or a sequencing process.

[0106] Aspect 6: A processor-implemented method as in any of the preceding aspects, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and the processor-implemented method further includes: assigning nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set of the one or more symmetric node sets.

[0107] Aspect 7: The processor-implemented method of any of the preceding aspects, wherein the representation comprises a computation graph.

[0108] Aspect 8: The processor-implemented method of any of the preceding aspects, wherein the representation comprises a computation graph, and the method further comprises: generating, via a processing device, the one or more of an order or a schedule for executing nodes of the computation graph.

[0109] Aspect 9: An apparatus comprising: a memory; and at least one processor coupled to the memory, the at least one processor configured to: receive a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; determine one or more symmetric sets of nodes based on a set of attributes of each node or one or more of the connectivity of the nodes via the plurality of edges; and generate one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0110] Aspect 10: An apparatus as in Aspect 9, wherein the at least one processor is further configured to: assign at least one attribute to each node; determine a node embedding for each node based on an aggregation of attributes of one or more of the predecessor nodes and / or successor nodes; and assign the nodes to one of the one or more symmetric node sets based on the determined node embeddings.

[0111] Aspect 11: The apparatus of aspect 9 or 10, wherein the at least one processor is further configured to: determine the one or more symmetric node sets based on a search space size and a degree of symmetry.

[0112] Aspect 12: The apparatus of any of Aspects 9-11, wherein the at least one processor is further configured to iteratively update the node embedding of each node based on other node embeddings.

[0113] Aspect 13: The apparatus of any of Aspects 9-12, wherein the one or more of the order or schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

[0114] Aspect 14: An apparatus as in any of Aspects 9-13, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and wherein the at least one processor is further configured to: assign nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set of the one or more symmetric node sets.

[0115] Aspect 15: The apparatus of any one of Aspects 9-14, wherein the representation comprises a computation graph.

[0116] Aspect 16: The apparatus of any of Aspects 9-15, wherein the representation comprises a computation graph, and the at least one processor is further configured to: generate, via the processing device, the one or more of an order or a schedule for executing nodes of the computation graph.

[0117] Aspect 17: An apparatus comprising: a device for receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; a device for determining one or more symmetric sets of nodes based on a set of attributes of each node or one or more of the connectivity of the nodes via the plurality of edges; and a device for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0118] Aspect 18: The apparatus of Aspect 17 further comprises: a device for assigning at least one attribute to each node; a device for determining a node embedding for each node based on an aggregation of attributes of one or more of the predecessor nodes and / or successor nodes; and a device for assigning these nodes to one of the one or more symmetric node sets based on the determined node embedding.

[0119] Aspect 19: The apparatus of aspect 17 or 18, further comprising: means for determining the one or more symmetric node sets based on the search space size and the degree of symmetry.

[0120] Aspect 20: The apparatus of any of Aspects 17-19, further comprising: means for iteratively updating the node embedding of each node based on the other node embeddings.

[0121] Aspect 21: The apparatus of any of Aspects 17-20, wherein the one or more of the order or schedule in each symmetry set is determined based on at least one of a graph layout process, a tiebreaker process, or a sequencing process.

[0122] Aspect 22: An apparatus as in any of Aspects 17-21, wherein the set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and the apparatus further includes: a device for assigning nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set in the one or more symmetric node sets.

[0123] Aspect 23: The apparatus of any of Aspects 17-22, wherein the representation comprises a computation graph.

[0124] Aspect 24: The apparatus of any of Aspects 17-23, wherein the representation comprises a computation graph, and the apparatus further comprises: means for generating, via a processing device, the one or more of an order or a schedule for executing nodes of the computation graph.

[0125] Aspect 25: A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising: program code for receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; program code for determining one or more symmetric sets of nodes based on one or more of a set of attributes of each node or the connectivity of the nodes via the plurality of edges; and program code for generating one or more of an order or schedule for executing the nodes based on the one or more symmetric sets of nodes.

[0126] Aspect 26: Non-transitory computer-readable medium as in Aspect 25, wherein the program code further comprises: program code for assigning at least one attribute to each node; program code for determining a node embedding for each node based on an aggregation of attributes of one or more of the predecessor nodes and / or successor nodes; and program code for assigning the nodes to one of the one or more symmetric node sets based on the determined node embedding.

[0127] Aspect 27: The non-transitory computer-readable medium of aspect 25 or 26, wherein the program code further comprises: program code for determining the one or more symmetric node sets based on a search space size and a degree of symmetry.

[0128] Aspect 28: The non-transitory computer-readable medium of any of Aspects 25-27, wherein the program code further comprises: program code for iteratively updating the node embedding of each node based on the other node embeddings.

[0129] Aspect 29: The non-transitory computer-readable medium of any of Aspects 25-28, wherein the one or more of the order or schedule in each symmetric set is determined based on at least one of a graph layout process, a tiebreaker process, or a sequencing process.

[0130] Aspect 30: Non-transitory computer-readable medium as in any of Aspects 25-29, wherein the set of attributes comprises one or more of path length, node type, output tensor size, or node execution duration, and the non-transitory computer-readable medium further comprises: program code for assigning nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set of the one or more symmetric node sets.

[0131] In one aspect, the receiving means, determining means, and / or generating means may be the GPU 104, program memory associated with the GPU 104, the fully connected layer 362, the NPU 428, and / or the routing connection processing unit 216 configured to perform the recited functions. In another configuration, the aforementioned means may be any module or any equipment configured to perform the functions recited by the aforementioned means.

[0132] The various operations of the methods described above may be performed by any suitable device capable of performing the corresponding functions. These devices may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally speaking, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0133] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or another data structure), ascertaining, and the like. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determine" may include resolving, selecting, choosing, establishing, and the like.

[0134] As used, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0135] The various illustrative logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0136] The steps of the method or algorithm described in conjunction with the present disclosure can be implemented directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module can reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module can include a single instruction or many instructions and can be distributed over several different code segments, distributed between different programs, and distributed across multiple storage media. A storage medium can be coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium can be integrated into a processor.

[0137] The disclosed methods include one or more steps or actions for achieving the described method. These method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of the specific steps and / or actions may be modified without departing from the scope of the claims.

[0138] The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system in a device. The processing system may be implemented using a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnecting buses and bridges. The bus may link together various circuits including a processor, a machine-readable medium, and a bus interface. The bus interface may be used to connect a network adapter, etc., to the processing system via the bus. The network adapter may be used to implement signal processing functions. For certain aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, and the like, which are well known in the art and will not be described further.

[0139] The processor may be responsible for managing the bus and general processing, including executing software stored on a machine-readable medium. The processor may be implemented with one or more general and / or special processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuit systems capable of executing software. Software should be broadly interpreted to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or other. As an example, a machine-readable medium may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be implemented in a computer program product. The computer program product may include packaging material.

[0140] In a hardware implementation, the machine-readable medium can be a part of the processing system that is separate from the processor. However, as will be readily appreciated by those skilled in the art, the machine-readable medium or any part thereof can be external to the processing system. As an example, the machine-readable medium can include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all of which can be accessed by the processor via a bus interface. Alternatively or additionally, the machine-readable medium or any part thereof can be integrated into the processor, as may be the case with a cache and / or general register file. Although the various components discussed can be described as having specific locations, such as local components, they can also be configured in various ways, such as with certain components being configured as part of a distributed computing system.

[0141] The processing system can be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable medium, all linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system can include one or more neuromorphic processors for implementing the described neuron model and neural system model. As another alternative, the processing system can be implemented using an application-specific integrated circuit (ASIC) with a processor, bus interface, user interface, supporting circuitry, and at least a portion of the machine-readable medium integrated into a single chip, or using one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gating logic, discrete hardware components, or any other suitable circuitry, or any combination of circuits capable of performing the various functionalities described throughout this disclosure. Depending on the specific application and the overall design constraints imposed on the overall system, one skilled in the art will recognize how to best implement the functionality described with respect to the processing system.

[0142] The machine-readable medium may include several software modules. These software modules include instructions that, when executed by a processor, cause a processing system to perform various functions. These software modules may include a transmitting module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. As an example, when a triggering event occurs, a software module may be loaded from a hard drive into RAM. During execution of the software module, the processor may load some instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general register file for execution by the processor. When describing the functionality of a software module below, it will be understood that such functionality is implemented by the processor when the processor executes instructions from the software module. In addition, it should be appreciated that various aspects of the present disclosure result in improvements to the functionality of a processor, computer, machine, or other system that implements such aspects.

[0143] If implemented in software, each function may be stored as one or more instructions or codes on or transmitted by a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared (IR), radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Thus, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Additionally, for other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0144] Thus, some aspects may include a computer program product for performing the operations presented. For example, such a computer program product may include a computer-readable medium having stored (and / or encoded) thereon instructions, which are executable by one or more processors to perform the described operations. For some aspects, the computer program product may include packaging materials.

[0145] Furthermore, it should be appreciated that modules and / or other appropriate means for performing the described methods and techniques can be downloaded and / or otherwise obtained by a user terminal and / or base station, where applicable. For example, such a device can be coupled to a server to facilitate the transfer of the means for performing the described methods. Alternatively, the various methods described can be provided via a storage device (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or floppy disk, etc.), such that once the storage device is coupled to or provided to a user terminal and / or base station, the device can obtain the various methods. Furthermore, any other suitable technology suitable for providing the described methods and techniques to a device can be utilized.

[0146] It will be understood that the claims are not limited to the precise configuration and components illustrated above. Various changes, substitutions and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A processor-implemented method, comprising: receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; determining one or more symmetric node sets based on one or more of a set of attributes of each node or connectivity of the nodes via the plurality of edges; and One or more of an order or a schedule for executing the nodes is generated based on the one or more symmetric sets of nodes.

2. The processor-implemented method of claim 1 , further comprising: Assign at least one attribute to each node; determining a node embedding for each node based on an aggregation of attributes of one or more of a predecessor node and / or a successor node; as well as The node is assigned to one of the one or more symmetric node sets based on the determined node embedding.

3. The processor-implemented method of claim 2 , further comprising: The one or more symmetric node sets are determined based on the search space size and the degree of symmetry.

4. The processor-implemented method of claim 2 , further comprising: The node embedding of each node is iteratively updated based on the other node embeddings.

5. The processor-implemented method of claim 1 , wherein: The one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

6. The processor-implemented method of claim 1 , wherein: The set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and the processor-implemented method further includes assigning nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set of the one or more symmetric node sets.

7. The processor-implemented method of claim 1 , wherein: The representation includes a computation graph.

8. The processor-implemented method of claim 1 , wherein: The representation includes a computation graph, and the processor-implemented method further includes generating, via a processing device, the one or more of the order or the schedule for executing the nodes of the computation graph.

9. A device comprising: Memory; as well as at least one processor coupled to the memory, the at least one processor configured to: receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; determining one or more symmetric node sets based on one or more of a set of attributes of each node or connectivity of the nodes via the plurality of edges; and One or more of an order or a schedule for executing the nodes is generated based on the one or more symmetric sets of nodes.

10. The apparatus of claim 9, wherein the at least one processor is further configured to: Assign at least one attribute to each node; determining a node embedding for each node based on an aggregation of attributes of one or more of a predecessor node and / or a successor node; and The node is assigned to one of the one or more symmetric node sets based on the determined node embedding.

11. The device according to claim 10, wherein The at least one processor is further configured to determine the one or more symmetric node sets based on a search space size and a degree of symmetry.

12. The device according to claim 10, wherein The at least one processor is further configured to iteratively update the node embedding of each node based on other node embeddings.

13. The apparatus of claim 9, wherein: The one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

14. The apparatus of claim 9, wherein: The set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and wherein the at least one processor is further configured to assign nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set of the one or more symmetric node sets.

15. The apparatus of claim 9, wherein: The representation includes a computation graph.

16. The apparatus of claim 9, wherein: The representation includes a computation graph, and the at least one processor is further configured to generate, via a processing device, the one or more of the order or the schedule for executing the nodes of the computation graph.

17. An apparatus comprising: means for receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; means for determining one or more symmetric sets of nodes based on one or more of a set of attributes of each node or connectivity of the nodes via the plurality of edges; as well as Means for generating one or more of an order or a schedule for executing the nodes based on the one or more symmetric sets of nodes.

18. The apparatus of claim 17, further comprising: means for assigning at least one attribute to each node; means for determining a node embedding for each node based on an aggregation of attributes of one or more of a predecessor node and / or a successor node; as well as Means for assigning the node to one of the one or more symmetric node sets based on the determined node embedding.

19. The apparatus of claim 18, further comprising: Means for determining the one or more symmetric node sets based on a search space size and a degree of symmetry.

20. The apparatus of claim 18, further comprising: Means for iteratively updating the node embedding of each node based on the embeddings of other nodes.

21. The apparatus of claim 17, wherein: The one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

22. The apparatus of claim 17, wherein: The set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and the apparatus further includes: a device for assigning nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to a symmetric set of the one or more symmetric node sets.

23. The apparatus of claim 17, wherein: The representation includes a computation graph.

24. The apparatus of claim 17, wherein: The representation includes a computation graph, and the apparatus further includes means for generating, via a processing device, the one or more of the order or the schedule for executing the nodes of the computation graph.

25. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising: program code for receiving a representation of an artificial neural network (ANN), the representation comprising a plurality of nodes coupled via a plurality of edges; program code for determining one or more symmetric sets of nodes based on one or more of a set of attributes of each node or connectivity of the nodes via the plurality of edges; as well as Program code is provided for generating one or more of an order or a schedule for executing the nodes based on the one or more symmetric sets of nodes.

26. The non-transitory computer readable medium of claim 25, wherein the program code further comprises: program code for assigning at least one attribute to each node; program code for determining a node embedding for each node based on an aggregation of attributes of one or more of a predecessor node and / or a successor node; as well as Program code for assigning the node to one of the one or more symmetric node sets based on the determined node embedding.

27. The non-transitory computer readable medium of claim 26, wherein: The program code further includes program code for determining the one or more symmetric node sets based on a search space size and a degree of symmetry.

28. The non-transitory computer readable medium of claim 26, wherein: The program code further includes program code for iteratively updating the node embedding of each node based on other node embeddings.

29. The non-transitory computer readable medium of claim 25, wherein: The one or more of the order or the schedule in each symmetric set is determined based on at least one of a graph layout process, a tie-breaking process, or a sequencing process.

30. The non-transitory computer readable medium of claim 25, wherein: The set of attributes includes one or more of path length, node type, output tensor size, or node execution duration, and the non-transitory computer-readable medium further includes: program code for assigning nodes having one or more of the same path length, the same node type, the same output tensor size, or the same node execution duration to symmetric sets in the one or more symmetric node sets.

Images