Optimizing machine learning model performance

Abstract

Certain aspects of the present disclosure provide techniques for receiving data defining a neural network; analyzing the data to determine depth-first cut points for a depth-first traversal portion of a full network traversal; performing a depth-first traversal for the depth-first portion of the full network traversal; and performing a layer-based traversal for the layer-based portion of the full network traversal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 905,541, filed on June 18, 2020, which claims priority to and the benefits of U.S. Provisional Patent Application No. 62 / 863,099, filed on June 18, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various aspects of the present disclosure relate to optimizing machine learning model performance. Background Art

[0004] Machine learning typically produces a trained model (e.g., an artificial neural network, tree, or other structure) that represents a generalized fit to a training data set known a priori. Applying the trained model to new data produces inferences, which can be used to gain insights into the new data. In some cases, applying a model to new data is described as "running inferences" on the new data.

[0005] Many types of computing hardware blocks can be used to run inference, including, for example: central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), neural processing units (NPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and other custom hardware.

[0006] Key performance indicators (KPIs) can quantify the performance of a trained model, such as the "quality" and "efficiency" of inferences from the trained model. For example, the accuracy of inferences, the speed of inferences, the size of inferences, the power consumed by the device performing the inferences, the hardware utilization and bandwidth of the device performing the inferences, and other items are examples of KPIs.

[0007] Typically, optimization of one KPI is at the expense of one or more other KPIs. Therefore, there is a need for systems and methods for improving trained model performance so that KPIs can be improved simultaneously. Summary of the invention

[0008] In a first aspect, an embodiment described herein includes a method comprising: receiving data defining a neural network; analyzing the data to determine depth-first cut points for a depth-first traversal portion of a full network traversal; performing a depth-first traversal for the depth-first portion of the full network traversal; and performing a layer-based traversal for a layer-based portion of the full network traversal.

[0009] In a second aspect, an embodiment described herein includes a method comprising: receiving data defining a neural network; receiving hardware characteristic data defining a plurality of hardware resources, the hardware characteristic data including a local memory capacity; analyzing the data to determine a cut point for a depth-first traversal portion of a full network traversal; determining a first cut point based at least in part on the data and the local memory capacity, wherein the first cut point is located at a maximum depth of the neural network given the local memory capacity; determining a second cut point, wherein the second cut point is shallower than the first cut point, and wherein the second cut point increases hardware parallelism compared to the first cut point; and performing a depth-first traversal for the depth-first portion of the full network traversal using two or more of the plurality of hardware resources and the second cut point.

[0010] In a third aspect, an embodiment described herein includes a method comprising: during a first time interval, performing a computation phase of a first inference on a first hardware resource; during a second time interval, performing a transmission phase of the first inference on a second hardware resource; during the second time interval, performing a computation phase of a second inference on the first hardware resource; and during a third time interval, performing the transmission phase of the second inference on the second hardware resource.

[0011] Further embodiments relate to an apparatus configured to perform the methods described herein, and a non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a device, cause the device to perform the methods described herein.

[0012] The following description and the associated drawings set forth in detail certain illustrative features of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings depict certain aspects of one or more embodiments and therefore should not be considered to limit the scope of the disclosure.

[0014] Figures 1A-1D Depicted are examples of various types of neural networks.

[0015] Figure 2 is a block diagram illustrating an exemplary deep convolutional network.

[0016] Figure 3A Depicted is an example of a traditional depth-first network traversal strategy.

[0017] Figure 3B Depicted is an example of a hybrid depth-first and full-layer traversal strategy.

[0018] Figure 4A and4B Depicted is an example of a traditional hardware parallelization scheme for convolution computation.

[0019] Figure 4C and 4D Depicted is an example of an improved hardware parallelization scheme for convolution computations.

[0020] Figure 5A and 5B Depicts an example of depth-first cut optimization for a hardware parallelization scheme for convolution computations.

[0021] Figure 6 Depicts an example of batch optimization for improving hardware parallelism.

[0022] Figure 7 Depicted is an example method for performing a hybrid depth-first and layer-based network traversal.

[0023] Figure 8 Depicted are example methods for parallelizing hardware resources during network traversal.

[0024] Fig. 9 Depicts an example method for interleaving batches of neural network processing.

[0025] Fig.10 An example implementation of a system on a chip (SOC) is shown.

[0026] Fig.11 Depicted is an example schematic diagram of a multi-processor processing system that may be implemented utilizing embodiments described herein.

[0027] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features disclosed in one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION

[0028] Aspects of the present disclosure provide apparatus, methods, processing systems, and computer-readable media for optimizing the performance of machine learning models, such as neural networks, in hardware.

[0029] Specifically, aspects provided herein allow for determining whether data between portions (e.g., layers) of a neural network model needs to be written to non-local memory (e.g., host system memory, such as DRAM, etc.) or whether it can be stored in local memory (e.g., tightly coupled memory), which improves access speed, reduces system data bus utilization, and reduces power consumption.

[0030] Aspects provided herein can improve depth-first network traversal of a neural network. In general, depth-first traversal refers to traveling through multiple layers of a neural network in a depth-first manner to minimize intermediate memory bandwidth. Depth-first traversal seeks to generate and consume intermediate neural network data in such a way that intermediate neural network data is not required to be unloaded to non-local memory and unnecessarily reloaded from non-local memory. More specifically, depth-first traversal seeks to minimize the following power-consuming and time-consuming operations: writing the activation outputs of one layer to a host processing system memory (e.g., DRAM) and then reading these outputs from the host processing system memory as inputs to subsequent layers. For example, U.S. patent application No. 16 / 443,695, entitled “Depth-First Convolution in Deep Neural Networks,” filed on June 17, 2019 (the entire contents of which are incorporated herein by reference), discloses methods for depth-first traversal.

[0031] In a conventional depth-first approach for processing neural network models, data between two different parts of a neural network model (including two different depth-first parts) is written to non-local memory (overflow) and then read back (fill) to local memory for processing. This process takes time, thus increasing processing latency and power, thereby increasing the overall power usage of the processing (and reducing power efficiency). Therefore, conventional depth-first processing of neural models cannot take advantage of the advantages of making depth-first segments shallower (if it allows the ability to save ("pin") data between depth-first segments). Aspects described herein take this benefit into account during processing (e.g., when determining cut points between model parts) to increase speed (e.g., by reducing memory latency) and reduce power usage (e.g., by reducing writing to memory and reading from memory). Notably, the aspects described herein are applicable in multi-stage neural network models using different batching schemes.

[0032] Deep Neural Networks and Deep Learning

[0033] Deep learning architectures can perform complex tasks (such as object recognition) by learning to represent inputs at successively higher levels of abstraction in each layer, thereby building useful feature representations of the input data. In this way, deep learning solves a major bottleneck of traditional machine learning. Before the advent of deep learning, machine learning methods for tasks may rely heavily on manually designed features, possibly combined with shallow classifiers. A shallow classifier can be a two-class linear classifier, for example, in which a weighted sum of input values ​​(e.g., input vector components) can be compared to a threshold to predict which category the input belongs to. Manually designed features can be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, deep learning architectures can learn to represent features similar to those that a human engineer might design, but through training. In addition, deep networks can learn to represent and recognize new types of features that humans may not have considered.

[0034] Deep learning architectures can learn hierarchies of features. For example, if given with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if given with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, which takes the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Higher layers can learn to recognize common visual objects or spoken phrases.

[0035] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motor vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to recognize cars, trucks, and airplanes.

[0036] Neural networks are designed with multiple connection patterns. For example, in a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. Neural networks can also have loops or feedback (also called top-down) connections. In a loop connection, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. The loop architecture can help identify patterns that span more than one block of input data that is delivered to the neural network in sequence. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections can be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input.

[0037] The connections between layers of a neural network can be fully connected or partially connected. Figure 1A An example of a fully connected neural network 102 is shown. In a fully connected neural network 102, a neuron in a first layer may transmit its output to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer.

[0038] Figure 1B An example of a locally connected neural network 104 is shown. In the locally connected neural network 104, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 104 may be configured such that each neuron in a layer will have the same or similar connection pattern, but with connection strengths with different values ​​(e.g., 110, 112, 114, and 116). The connectivity patterns of the local connections may result in spatially different receptive fields in higher layers because higher layer neurons in a given area may receive inputs that are tuned through training to properties of a limited portion of the total input to the network.

[0039] An example of a locally connected neural network is a convolutional neural network. Figure 1C An example of a convolutional neural network 106 is shown. The convolutional neural network 106 can be configured so that the connection strengths associated with the inputs to each neuron in the second layer are shared (e.g., 108). Convolutional neural networks can be well suited for problems in which the spatial location of the inputs is meaningful.

[0040] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basic prediction. If the input is first decomposed into multiple channels (such as the red, green, and blue channels of a color image), the convolutional network trained on this input can be thought of as three-dimensional, with two spatial dimensions along the axis along the image and the third dimension capturing color information. The output of the convolutional connection can be thought of as forming a feature map in the subsequent layer, where each element of the feature map receives input from a series of neurons in the previous layer and from each of the multiple channels. The values ​​in the feature map can be further processed using nonlinearities (e.g., rectification max(0,x)). The values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0041] One type of convolutional neural network is a deep convolutional network (DCN). A deep convolutional network (DCN) is a network of convolutional layers that is configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0042] The DCN can be a feed-forward network. In addition, as described above, the connections from neurons in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN can be used for fast processing. For example, the computational burden of the DCN can be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0043] Figure 1D A detailed example of a DCN 100 is shown, which is designed to recognize visual features based on an image 126 input from an image capture device 130 (such as an onboard camera). The DCN 100 of the current example can be trained to recognize traffic signs and numbers provided on traffic signs. Of course, the DCN 100 can be trained for other tasks, such as recognizing lane markings or recognizing traffic lights. These are just some example tasks, and many other tasks are possible.

[0044] The DCN 100 may be trained using supervised learning. During training, an image (such as an image 126 of a speed limit sign) may be presented to the DCN 100, and a forward pass may then be calculated to produce an output 122. The DCN 100 may include a feature extraction portion and a classification portion. Upon receiving the image 126, a convolutional layer 132 may apply a convolutional kernel (not shown) to the image 126 to generate a first feature map set 118. As an example, the convolutional kernel used for the convolutional layer 132 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first feature map set 118, four different convolutional kernels are applied to the image 126 at the convolutional layer 132. A convolutional kernel may also be referred to as a filter or a convolutional filter.

[0045] The first feature map set 118 may be subsampled by a maximum pooling layer (not shown) to generate a second feature map set 120. The maximum pooling layer reduces the size of the first feature map set 118. That is, the size of the second feature map set 120 (such as 14x14) is smaller than the size of the first feature map set 118 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second feature map set 120 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent feature map sets (not shown).

[0046] exist Figure 1D In the example of , the second feature map set 120 is convolved to generate a first feature vector 124. In addition, the first feature vector 124 is further convolved to generate a second feature vector 128. Each feature of the second feature vector 128 may include a number corresponding to a possible feature of the image 126 (e.g., "sign", "60", and "100"). A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. As such, the output 122 of the DCN 100 is the probability that the image 126 includes one or more features.

[0047] In this example, the probabilities for "logo" and "60" in output 122 are higher than the probabilities for other items of output 122 (e.g., "30", "40", "50", "70", "80", "90", and "100"). Before training, the output 122 produced by DCN 100 may be incorrect. Therefore, the error between the output 122 and the target output can be calculated. The target output is the ground truth of image 126 (e.g., "logo" and "60"). Then, the weights of DCN 100 can be adjusted so that the output 122 of DCN 100 is more closely aligned with the target output.

[0048] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0049] In practice, the error gradient of the weights can be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level. After learning, new images may be presented to the DCN 100, and a forward pass through the network may produce an output 122 that may be considered an inference or prediction of the DCN.

[0050] Finally, a deep belief network (DBN) is a probabilistic model comprising multiple layers of hidden nodes. DBN can be used to extract a hierarchical representation of a training data set. DBN can be obtained by superimposing the layers of a restricted Boltzmann machine (RBM). RBM is a type of artificial neural network that can learn probability distributions through an input set. Since RBM can learn probability distributions without information about the categories to which each input should be classified, RBM is often used in unsupervised learning. Using a mixed unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained in an unsupervised manner and can act as a feature extractor, while the top RBM can be trained in a supervised manner (based on the joint distribution of the input and target categories from the previous layer) and can act as a classifier.

[0051] Figure 2 is a block diagram illustrating an exemplary deep convolutional network 250. The deep convolutional network 250 may include multiple different types of layers based on connectivity and weight sharing. Figure 2 As shown, the deep convolution network 250 includes convolution blocks 254A and 254B. Each of the convolution blocks 254A and 254B may be configured to have a convolution layer (CONV) 256, a normalization layer (LNorm) 258, and a maximum pooling layer (MAX POOL) 260.

[0052] The convolution layer 256 may include one or more convolution filters that may be applied to the input data 252 to generate a feature map. Although only two convolution blocks 254A and 254B are shown, the present disclosure is not limited thereto, and on the contrary, any number of convolution blocks (e.g., blocks 254A and 254B) may be included in the deep convolution network 250, depending on design preferences. The normalization layer 258 may normalize the output of the convolution filter. For example, the normalization layer 258 may provide whitening or lateral suppression. The maximum pooling 260 may provide down-sampling aggregation in space to obtain local invariance and dimensionality reduction.

[0053] For example, a parallel filter bank of a deep convolutional network can be loaded into a processing device (such as Fig.10described) to achieve high performance and low power consumption.

[0054] The deep convolutional network 250 may also include one or more fully connected layers, such as layer 262A (labeled as “FC1”) and layer 262B (labeled as “FC2”). The deep convolutional network 250 may also include a logistic regression (LR) layer 264. Between each layer 256, 258, 260, 262, 264 of the deep convolutional network 250 are weights (not shown) to be updated. The output of each of these layers (e.g., 256, 258, 260, 262, 264) may serve as the input of a subsequent layer of these layers (e.g., 256, 258, 260, 262, 264) of the deep convolutional network 250 to learn a hierarchical feature representation from the input data 252 (e.g., image, audio, video, sensor data, and / or other input data) provided at the initial convolutional block in the convolutional block 254A. The output of the deep convolutional network 250 is a classification score 266 for the input data 252. The classification score 266 may be a set of probabilities, where each probability is a probability that the input data includes a feature from a set of features.

[0055] Model Inference Performance Strategy

[0056] Running inference using a trained machine learning model (such as the one described above) on a processing system can involve a series of steps that can be modified for optimization. For example, in the context of a neural network-based model, many layers of the model may need to be processed, and the order and manner in which the layers are processed may be manipulated to address different KPIs, such as speed, accuracy, power usage, etc.

[0057] A strategy known as layer-based mapping maps each layer of a neural network to a hardware resource so that the resource fully processes that layer before the next layer. Unfortunately, this strategy can require a lot of bandwidth because the relevant data may need to be loaded and unloaded repeatedly throughout the process of processing each layer. For example, in a network that uses large images as input data, intermediate data (e.g., activations and / or feature maps) may not be placed in local memory (e.g., local to the hardware resource), and therefore they must be unloaded to another memory, such as host system memory (e.g., DDR). This problem is particularly prominent for convolutional neural networks, where overlapping data is used for convolution. Significant memory bandwidth increases power usage, which can be a significant obstacle in mobile device applications. In addition, significant memory bandwidth may block various memory paths, causing other processes on the host processing system to slow down or pause.

[0058] In addition, assigning an entire layer to a single hardware resource does not maximize the parallelism of the hardware resource, which may result in fewer inferences per unit time (an example of a KPI). Modern electronic devices typically have multiple hardware resources (such as CPUs, GPUs, DSPs, NPUs, FPGAs, ASICs, and other custom hardware (and in some cases, multiple of the higher-level hardware resources described above, such as multiple matrix processors on a given ASIC)) that can be used to run inferences on the model. Better performance can be achieved based on utilizing these resources simultaneously through improved parallelization.

[0059] As described above, another strategy for traversing a neural network is a depth-first traversal, which typically advances through multiple layers of a neural network depth-first to minimize the time and power associated with moving data out to host processing memory (e.g., spilling data) and then moving data back from host processing memory between layers (e.g., filling data). However, when the amount of data generated (e.g., intermediate data such as activations and feature maps) exceeds the local memory capacity of the processing hardware, necessitating the need to write the data to another non-local memory (e.g., DDR of the host processing system) and later retrieve the data from it, maximizing the depth in the depth-first strategy may not minimize the bandwidth used. Furthermore, maximizing the depth may also have a negative impact on inferences per second.

[0060] Another strategy is batching, which is used to reduce the memory retrieval overhead compared to the computation used for inference. For example, if two inferences are computed at once, batching saves the bandwidth required for weight retrieval, because a single retrieval of weights can be used for both inferences. However, in some networks, this strategy does not produce the best KPIs, such as inferences per second (IPS).

[0061] The embodiments described herein improve the performance based on layer, depth-first and batch processing strategies, so that model performance can be improved. Specifically, the methods described herein can improve some KPIs (e.g., IP and memory bandwidth usage) without sacrificing other KPIs.

[0062] Hybrid depth-first and full-layer computation traversal

[0063] A conventional depth-first traversal of a neural network maximizes the subgraph sizing (i.e., the number of layers processed) and "overflows" data at cut points in the neural network architecture into non-local memory when the layer data does not fit into a local memory (such as a tightly coupled memory (TCM), such as an SRAM TCM) associated with a hardware resource that performs the processing of the neural network portion (e.g., a subgraph). The overflowed data is then read back into the local memory for use in processing the next portion (e.g., a subgraph) of the neural network. Figure 3AAn example of such a strategy, which may be referred to as a greedy strategy, is depicted.

[0064] Specifically, in Figure 3A , assuming 4MB local memory, the depth-first traversal will produce a cut at 302 that defines the extent of subgraph 304A, and data C (value 0.5MB) will "spill over" into non-local memory, such as DDR memory. In this example, data C overflows at cut point 302 because weights A+B+C+D (a total of 6MB) cannot fit into local memory (4MB), and similarly, weights A+B+C+data C will fill the entire local memory (4MB) and leave no room for convolution data (e.g., convolution cone data).

[0065] Data C is later "filled" back into local memory for convolution D in subgraph 304B. Note that spilling and filling back and forth requires twice the memory bandwidth for the data size, so in this case, spilling 0.5MB of data and then filling 0.5MB of data results in 1MB of memory bandwidth usage.

[0066] Similarly, for a layer-based traversal with the same 4MB local memory, data A will need to be overflowed and filled because weight A (1MB) and data A (4MB) exceed the 4MB local memory capacity.

[0067] In some implementations, the local memory may be a tightly coupled memory (TCM), which is typically a small dedicated memory area very close to the processor. The benefit of the TCM being close to the processor is that the processor can access the TCM every cycle, and unlike non-local memory, no cache is involved, which makes all memory accesses predictable. In addition, extracting data from the TCM is significantly more power efficient than extracting data from non-local memory, such as system memory (e.g., DDR).

[0068] Figure 3B A hybrid depth-first and full-level traversal scheme is depicted, where the cut point 312 is selected so that a depth-first traversal can be applied to the first subgraph 314A, and a full-level traversal can be applied to the second subgraph 314B. In this way, overflow and fill (as with respect to Figure 3A described), minimizing memory bandwidth. In other words, Figure 3B An example of selecting cut points is depicted so that model data can be stored in local memory during processing (thereby avoiding memory bandwidth utilization), rather than selecting cut points that only maximize the achievable depth in the model based on available memory. This improves memory bandwidth utilization.

[0069] Specifically, in this hybrid scheme, data B (0.5MB), weight A (1MB), weight B (1MB), and convolution cone data (<0.5MB) (not depicted) are stored in local memory instead of writing these data out to non-local memory. Therefore, in this hybrid scheme, the amount of convolution cone data can be considered when determining the cut point. Keeping these data in local memory saves the bandwidth of overflowing data to non-local memory (such as DDR memory).

[0070] The size of the convolution cone is based on many factors. For example, in order to generate the smallest size patch at the end of the convolution cone, you need to generate larger patches at the beginning of the convolution cone. The smallest patch size can be based on the hardware configuration (for example, a piece of hardware may be very inefficient if the computation is smaller than a certain number of KB). The larger patch size of the convolution cone through the layers is based on the nature of the computation. For example, striding two convolutions can increase the size by a factor of about 2.

[0071] After the cut point 312, the last two layers, including convolutions C and D, are processed in a layer-based manner, thus being a "hybrid" approach. A layer-based approach is beneficial here because weights C + data C = 2MB and weights D + data D = 3MB, both of which are below the local memory capacity of 4MB. Therefore, the total data in convolutions C and D is small enough to avoid any further overflow and fill, and thereby improve memory bandwidth. In other words, by considering smaller depth subgraphs, data (e.g., weights and data) can be "pinned" to local memory (e.g., TCM) between subgraphs, so overflow and fill from non-local memory is not necessary. It should be noted that the shortest subgraph can be a layer, but this does not need to be the case.

[0072] To implement this hybrid approach, the compiler may receive neural network data and hardware characteristic data (eg, including local memory size) and use both data to select a cut point 312 .

[0073] Parallelize multiple hardware computing resources

[0074] Traditional depth-first traversal also maximizes the size of neural network model subgraphs without considering balancing resource utilization between subgraphs, and does not run multiple parts of subgraphs in parallel (e.g., convolution cone parallelization). Similarly, traditional layer-based traversal does not attempt to simultaneously compute the layers of the neural network model, which may be mapped to different processing hardware resources.

[0075] An improvement over traditional approaches is to process multiple layers of a neural network model at a time, and select those layers with consideration for balancing hardware resource utilization between model subgraphs and using additional local memory to achieve convolution cone parallelization (and thus potentially reduce subgraph size).

[0076] For example, Figure 4A depicts an example convolutional neural network processing scheme, and Figure 4B Describes the use Figure 4A FIG. 1 is a timing diagram of the processing steps in FIG. 1 . In this example, convolution is performed on hardware resource X (HW X) and pooling is performed on hardware resource Y (HW Y).

[0077] It is worth noting that without exploiting hardware parallelization, the traditional depth-first scheme will not select the Figure 4B 402. As depicted, the completion time for this example is at time 402.

[0078] In contrast, Figure 4C An example is depicted where the cut point 408 is selected so as to have enough local memory for convolution cone parallelization (e.g., for convolution cones 1 and 2 as indicated). Figure 4D As shown, the optimization results in an improved completion time 404, which represents an interval improvement 406 (e.g., Figure 4B As depicted, the conventional method proceeds until time 402).

[0079] Therefore, allowing multiple parallel convolution cones to be traversed can improve hardware parallelism, which in turn reduces completion time.

[0080] Figure 5A and 5B Another example is depicted that compares hardware parallelization with traditional depth-first traversal cut selection and compares hardware parallelization with selected cut selection to maximize parallelization. In this example, hardware X (HW X) is mapped for convolution computation, while hardware resource Y (HW Y) is mapped for pooling.

[0081] A conventional depth-first traversal may select a cut point 502 that represents the deepest portion of the graph that can be reached given the available resources (e.g., as described above with respect to Figure 3A described). Figure 5B The timing diagram in shows that this strategy results in a completion time of 508 for two example convolution cones (cones 1A and 2A).

[0082] In contrast, cut selection 504 maximizes hardware parallelism for a given hardware mapping and results in improved completion times 506 for convolution cones 1B and 2B. In other words, hardware parallelism is maximized by modifying the traversal criterion in the case of cut 504 to take into account the distribution of multiple hardware blocks in each sub-map.

[0083] It should be noted that the representation of convolution cones 1A, 2A, 1B and 2B is simplified for convenience.

[0084] Inference batch interleaving

[0085] Figure 6 Examples of different batching strategies are depicted. Traditional batching in neural network inference is used to load weights once and reuse these weights for each inference (e.g., each batch) because the weights are common across all inferences. This can be seen by comparing timeline 602, which depicts two inferences in non-batch mode, and timeline 604, which depicts a timeline for two inferences batched in a traditional manner. As can be seen in timeline 604, the total time 610 for traditional batching is an improvement over the time 608 for non-batch mode. However, this improvement is limited to bandwidth savings due to the reduction in weights fetched from non-local memory. The computation phase takes as long as the non-batch mode in timeline 602, as depicted by range 616.

[0086] In contrast, as depicted by timeline 606, interleaving the batches to maximize hardware parallelism further improves the completion time to 612. In this case, different hardware resources may be utilized to perform the computations for batch 2 while the results for batch 1 are being moved to memory, as shown during interval 614.

[0087] Therefore, hardware parallelism can be improved by appropriately interleaving batches that take into account different mappings of hardware elements to compute-intensive or bandwidth-intensive tasks.

[0088] Example methods for optimizing machine learning model performance

[0089] Figure 7 An example method 700 of performing a hybrid depth-first and layer-based network traversal is depicted.

[0090] Method 700 begins at step 702, where data defining a neural network is received.

[0091] Method 700 then proceeds to step 704, where the data is analyzed to determine depth-first cut points for the depth-first traversal portion of the entire network traversal;

[0092] The method 700 then proceeds to step 706, where a depth-first traversal is performed for the depth-first portion of the entire network traversal; and

[0093] The method 700 then proceeds to step 708 where a layer-based traversal is performed for the layer-based portion of the overall network traversal.

[0094] In some embodiments, the method 700 further includes: during the depth-first portion of the overall network traversal, storing the weight data and output data of the depth-first traversal portion in a local memory.

[0095] In some embodiments of method 700, the depth-first cut point is selected based at least in part on a sum of a weight data size and an output data size for the depth-first traversal portion being less than a capacity of the local memory.

[0096] In some embodiments of the method 700, the local memory is a tightly coupled memory (TCM). For example, the TCM may be associated with a processor configured to perform a depth-first traversal and / or a layer-based traversal.

[0097] In some embodiments of method 700, performing a layer-based traversal for a layer-based portion of an overall network traversal does not require writing data to non-local memory.

[0098] For example, in some embodiments, performing a layer-based traversal for a layer-based portion of the entire network traversal includes: storing weight values ​​and input values ​​from a non-local memory in a local memory; performing processing of a first layer of the layer-based portion of the entire network traversal using the weight values ​​and the input values ​​to generate an output value; storing the output value in the local memory; and performing processing of a second layer of the layer-based portion of the entire network traversal using the weight values ​​and the output value.

[0099] The above article about Figure 3B An example of method 700 is depicted and described.

[0100] Figure 8 An example method 600 of parallelizing hardware resources during a network traversal is depicted.

[0101] Method 800 begins at step 802, where data defining a neural network is received.

[0102] The method 800 then proceeds to step 804, wherein hardware characteristic data defining a plurality of hardware resources is received, the hardware characteristic data including a local memory capacity;

[0103] The method 800 then proceeds to step 806 , where the data is analyzed to determine a cut point for the depth-first traversal portion of the entire network traversal;

[0104] The method 800 then proceeds to step 808, where a first cut point is determined based at least in part on the data and the local memory capacity, wherein the first cut point is located at a maximum depth of the neural network given the local memory capacity;

[0105] The method 800 then proceeds to step 810, where a second cut point is determined, where the second cut point is shallower than the first cut point, and where the second cut point increases hardware parallelism compared to the first cut point; and

[0106] The method 800 then proceeds to step 812 where a depth-first traversal is performed for a depth-first traversal portion of the entire network traversal using two or more of the plurality of hardware resources and the second cut point.

[0107] In some embodiments, method 800 further includes: during a depth-first portion of the overall network traversal, storing weight data and output data for the depth-first traversal portion in a local memory.

[0108] In some embodiments of method 800, the local memory is a tightly coupled memory (TCM). For example, the TCM may be associated with a processor configured to perform a depth-first traversal.

[0109] In some embodiments of method 800, performing a depth-first traversal for the depth-first traversal portion of the overall network traversal does not require writing data to non-local memory.

[0110] In some embodiments of method 800, the second cut point implements cone parallelization during the depth-first traversal for the depth-first portion of the overall network traversal.

[0111] The above article about Figure 4C , 4D , 5A and 5B depict and describe an example of method 800.

[0112] Fig. 9 An example method 900 for interleaving batches of neural network processing is depicted.

[0113] Method 900 begins at step 902 , where a first inferred computation phase is performed on a first hardware resource during a first time interval.

[0114] The method 900 then proceeds to step 904, wherein, during a second time interval, a first inferred transmission phase is performed on a second hardware resource;

[0115] The method 900 then proceeds to step 906, wherein, during a second time interval, a second inferred computation phase is performed on the first hardware resource; and

[0116] Method 900 then proceeds to step 908 where, during a third time interval, a second inferred transfer phase is performed on a second hardware resource.

[0117] In some embodiments of method 900 , the first hardware resource is optimized for inference computing.

[0118] In some embodiments of method 900, the second hardware resource is optimized for data transmission.

[0119] The above article about Figure 6 An example of method 900 is depicted and described.

[0120] Example processing system for optimizing machine learning model performance

[0121] Fig.10 An example implementation of a system on chip (SOC) 1000 according to certain aspects of the present disclosure is shown, and the SOC 1000 may include a central processing unit (CPU) 1002 or a multi-core CPU configured to execute a parallel Monte Carlo dropout function. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency band information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 1008, in a memory block associated with the CPU 1002, in a memory block associated with a graphics processing unit (GPU) 1004, in a memory block associated with a digital signal processor (DSP) 1006, in a memory block 1018, or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 1018.

[0122] The SOC 1000 may also include additional processing blocks customized for specific functions, such as a GPU 1004, a DSP 1006, a connection block 1010 (which may include fifth generation (5G) connections, fourth generation long term evolution (4G LTE) connections, Wi-Fi connections, USB connections, Bluetooth connections, etc.), and a multimedia processor 1012 (e.g., which may detect and recognize gestures). In one implementation, the NPU is implemented in the CPU 1002, the DSP 1006, and / or the GPU 1004. The SOC 1000 may also include a sensor processor 1014, an image signal processor (ISP) 1016, and / or a navigation module 1020 (which may include a global positioning system).

[0123] SOC 1000 can be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into CPU 1002 may include code for searching for the stored multiplication result in a lookup table (LUT) corresponding to the multiplication product of the input value and the filter weight. The instructions loaded into CPU 1002 may also include code for disabling the multiplier during the multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into CPU 1002 may include code for storing the calculated multiplication product of the input value and the filter weight when a lookup table missing of the multiplication product is detected.

[0124] SOC 1000 and / or its components may be configured to perform the methods described herein.

[0125] Fig.11 Depicted is an example schematic diagram of a multi-processor processing system 1100 that may be implemented utilizing embodiments described herein.

[0126] In this example, system 1100 includes processors 1101, 1103, and 1105, but in other examples, any number of single processors may be used. In addition, although similarly depicted, processors 1101, 1103, and 1105 may represent various different types of processors in electronic devices, such as CPUs, GPUs, DSPs, NPUs, etc. described herein.

[0127] Each of the processors 1101, 1103, and 1105 includes an instruction scheduler, various hardware subcomponents (e.g., hardware X, hardware Y, and hardware Z), and a local memory. As described above, the local memory may be a tightly coupled memory (TCM). It should be noted that the components of each of the processors 1101, 1103, and 1105 are shown as being the same in this example, and each of the processors may have different hardware configurations, different hardware elements, etc.

[0128] Each of the processors 1101 , 1103 , and 1105 is also in data communication with a global memory, such as a DDR memory, or other type of volatile working memory.

[0129] In some implementations, in a multi-processor processing system such as 1100, one of the processors may act as a master processor. For example, in this example, processor 1101 may be the master processor. The master processor may include a compiler that, when executed, may determine how a model such as a neural network will be processed by various components of the processing system 1100. For example, hardware parallelism may be achieved by mapping portions of the processing of the model to various hardware (e.g., hardware X, hardware Y, and hardware Z) within a given processor (e.g., processor 1101) and mapping portions of the processing of the model to other processors (e.g., processors 1103 and 1105) and their associated hardware, such as described above in Figure 4B , 4D , 5B and 6.

[0130] The previous description is provided to enable those skilled in the art to implement the various embodiments described herein. The examples discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made in the functions and arrangements of the elements discussed without departing from the scope of the present disclosure. Various processes or components may be omitted, replaced, or added to the various examples as appropriate. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, the features described with respect to some examples may be combined into some other examples. For example, a device may be implemented or a method may be implemented using any number of aspects set forth herein. In addition, the scope of the present disclosure is intended to cover such devices or methods implemented using other structures, functions, or structures and functions other than or different from the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0131] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0132] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover any combination of a, b, c, ab, ac, bc, and abc, as well as multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other ordering of a, b, and c).

[0133] As used herein, the term "determining" includes a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0134] The method disclosed herein includes one or more steps or actions for implementing the method. Without departing from the scope of the claims, these method steps and / or actions can be interchanged with each other. In other words, unless a specific order of steps or actions is specified, otherwise, without departing from the scope of the claims, the order and / or use of specific steps and / or actions can be modified. In addition, the various operations of the above-mentioned method can be performed by any appropriate unit capable of performing the corresponding functions. These units may include various hardware and / or software components and / or modules, including but not limited to: circuits, application specific integrated circuits (ASICs) or processors. Typically, in the presence of operations shown in the figure, those operations may have corresponding paired units plus functional components with similar numbers.

[0135] The following claims are not intended to be limited to the embodiments shown herein, but are given the full scope consistent with the language of the claims. Within the claims, unless explicitly stated as such, reference to an element in the singular form is not intended to mean "one and only one", but "one or more". Unless otherwise explicitly stated, the term "some" refers to one or more. No claim element is to be interpreted according to 35 U.S.C. § 112 (f), unless the element is explicitly recorded using the phrase "unit for...", or in the case of a method claim, the element is recorded using the phrase "step for...". All structural and functional equivalents of the elements of various aspects described throughout the present disclosure are expressly incorporated herein by reference, and are intended to be covered by the claims, and these structural and functional equivalents are known to or will be known later to those of ordinary skill in the art. In addition, no disclosed content herein is intended to be dedicated to the public, regardless of whether such disclosed content is explicitly recorded in the claims.

Claims

1. A method performed on a processing system comprising a processor and a local memory, the method comprising: receiving data defining a neural network; analyzing the data to determine depth-first cut points for a depth-first traversal portion of the entire network traversal; performing a depth-first traversal on the depth-first traversal portion of the entire network traversal; During the depth-first portion of the overall network traversal, storing in the local memory weight data and output data for the depth-first traversal portion; Determining the depth-first cut point based at least in part on a sum of a weight data size and an output data size for the depth-first traversal portion being less than a capacity of the local memory; as well as A layer-based traversal is performed for the layer-based portion of the entire network traversal.

2. The method according to claim 1, wherein: The local memory is a tightly coupled memory (TCM) associated with a processor configured to perform the depth-first traversal and the layer-based traversal.

3. The method according to claim 1, wherein: Performing a layer-based traversal for the layer-based portion of the entire network traversal includes: storing weight values ​​and input values ​​from a non-local memory in the local memory; performing processing of a first layer of the layer-based portion of the entire network traversal using the weight values ​​and the input values ​​to generate an output value; storing the output value in the local memory; and Processing of a second layer of the layer-based portion of the entire network traversal is performed using the weight values ​​and the output values.

4. The method according to claim 1, wherein: The depth-first cut point is determined by a compiler based on the capacity of the local memory.

5. A processing system comprising: a memory including computer executable instructions; Local storage; as well as A processor configured to execute the computer executable instructions and cause the processing system to perform the following operations: receiving data defining a neural network; analyzing the data to determine depth-first cut points for a depth-first traversal portion of the entire network traversal; performing a depth-first traversal on the depth-first traversal portion of the entire network traversal; During the depth-first portion of the overall network traversal, storing in the local memory weight data and output data for the depth-first traversal portion; Determining the depth-first cut point based at least in part on a sum of a weight data size and an output data size for the depth-first traversal portion being less than a capacity of the local memory; as well as A layer-based traversal is performed for the layer-based portion of the entire network traversal.

6. The processing system according to claim 5, wherein: The local memory is a tightly coupled memory (TCM) associated with a processor configured to perform the depth-first traversal and the layer-based traversal.

7. The processing system according to claim 5, wherein: To perform the layer-based traversal for the layer-based portion of the entire network traversal, the processor is further configured to: storing weight values ​​and input values ​​from a non-local memory in the local memory; performing processing of a first layer of the layer-based portion of the entire network traversal using the weight values ​​and the input values ​​to generate an output value; storing the output value in the local memory; as well as Processing of a second layer of the layer-based portion of the entire network traversal is performed using the weight values ​​and the output values.

8. The processing system according to claim 5, wherein: The depth-first cut point is determined by a compiler based on the capacity of the local memory.

9. A method comprising: receiving data defining a neural network; receiving hardware characteristic data defining a plurality of hardware resources, the hardware characteristic data including a local memory capacity; analyzing the data to determine cut points for a depth-first traversal portion of the overall network traversal; determining a first cut point based at least in part on the data and the local memory capacity, wherein the first cut point is located at a maximum depth of the neural network given the local memory capacity; determining a second cut point, wherein the second cut point is shallower than the first cut point, and wherein the second cut point increases hardware parallelism compared to the first cut point; and A depth-first traversal is performed for the depth-first portion of the entire network traversal using two or more of the plurality of hardware resources and the second cut point.

10. The method according to claim 9, further comprising: During the depth-first portion of the overall network traversal, weight data and output data for the depth-first traversal portion are stored in local memory.

11. The method according to claim 10, wherein: The local memory is a tightly coupled memory (TCM).

12. The method according to claim 9, wherein: Performing a depth-first traversal for the depth-first traversal portion of the entire network traversal does not require writing data to non-local memory.

13. The method according to claim 9, wherein: The second cut point enables cone parallelization during the depth-first traversal for the depth-first portion of the entire network traversal.

14. A processing system comprising: a memory including computer executable instructions; Local storage; as well as A processor configured to execute the computer executable instructions and cause the processing system to perform the method according to any one of claims 9-13.

Images