Processing for multiple input datasets

Abstract

To perform multi-layer neural network processing for multiple contexts.SOLUTION: A method comprise a computing engine using a first neural network layer to process first and second data sets to generate respective first layer outputs of the first and second data sets, using a second neural network layer to process the respective first layer outputs of the first and second data sets to generate respective second layer outputs of the first and second data sets, using the first neural network layer to process third and fourth data sets to generate respective first layer outputs of the third and fourth sets, using a second neural network layer to process the respective first layer outputs of the third and fourth data sets to generate respective second layer outputs of the third and fourth data sets, and using a third neural network to generate respective third layer outputs of the data sets.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001]

[0001] An artificial neural network is a computing system with an architecture based on biological neural networks. An artificial neural network can be trained to learn how to perform a specific computational task using training data. For example, the training data may include image data associated with a given object. A trained artificial neural network can process multiple images (or multiple sets of images) representing different contexts (e.g., captured in different places, at different times, by different people, and carrying different information) to determine whether the image or set of images contains image data of the given object. Summary of the Invention [Means for solving the problem]

[0002]

[0002] A neural network may include a set of processing nodes. Each processing node may process a portion of the input data to generate an output, and a final decision may be generated based on a combination of the outputs of the set of processing nodes. As part of its processing, each processing node may perform a set of arithmetic operations, such as floating-point multiplication and addition. The arithmetic operations of the processing nodes may be performed by circuitry and data paths optimized for high-speed arithmetic operations, such as a graphics processing unit (GPU). [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 illustrates an exemplary data processing device for processing images using the techniques disclosed herein. [Figure 2A] FIG. 1 is a simplified block diagram of a predictive model using techniques disclosed herein, in accordance with some aspects of the present disclosure. [Figure 2B]FIG. 1 is a simplified block diagram of a predictive model using techniques disclosed herein, in accordance with some aspects of the present disclosure. [Figure 2C] FIG. 1 is a simplified block diagram of a predictive model using techniques disclosed herein, in accordance with some aspects of the present disclosure. [Figure 2D] FIG. 1 is a simplified block diagram of a predictive model using techniques disclosed herein, in accordance with some aspects of the present disclosure. [Figure 3A] FIG. 3 is a simplified block diagram of some of the internal components of an apparatus for implementing the predictive model of FIG. 2 in accordance with some aspects of the present disclosure. [Figure 3B] FIG. 3 is a simplified block diagram of some of the internal components of an apparatus for implementing the predictive model of FIG. 2 in accordance with some aspects of the present disclosure. [Figure 3C] FIG. 3 is a simplified block diagram of some of the internal components of an apparatus for implementing the predictive model of FIG. 2 in accordance with some aspects of the present disclosure. [Figure 4] FIG. 1 illustrates an exemplary sequence for operating a computing system to perform multi-layer neural network processing of multiple data sets, in accordance with some aspects of the present disclosure. [Figure 5] FIG. 1 illustrates an exemplary sequence for operating a computing system to perform multi-layer neural network processing of multiple data sets, in accordance with some aspects of the present disclosure. [Figure 6] 1 is an exemplary flowchart of a method for operating a computing system to perform multi-layer neural network processing of multiple data sets in accordance with some aspects of the present disclosure. [Figure 7A] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 7B] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 7C] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 7D] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 7E] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 7F] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 7G] FIG. 1 illustrates an example of a scheduling scheme for scheduling multi-layer neural network processing of multiple data sets, according to some aspects of the present disclosure. [Figure 8] FIG. 10 is an exemplary flow diagram for scheduling operation of a computation engine according to some aspects of the present disclosure. [Figure 9] FIG. 1 illustrates an example of a computing device in accordance with some aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0004]

[0011] In the following description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0005]

[0012]

[0003] An embodiment of the present disclosure relates to a method for operating a computation engine to perform computations for a multi-layer neural network for multiple contexts. The multi-layer neural network may include a first neural network layer, a second neural network layer that receives data from the first neural network layer, and a third neural network layer that receives data from the second neural network layer. In one example, the computation engine may be configured as the first neural network layer to perform computations for the first context, then as the second neural network layer to perform computations for the first context, and may store the output of the second neural network layer computations for the first context. The computation engine may then switch back to the first neural network layer to perform computations for the second context, and then as the second neural network layer to perform computations for the second context. The output of the second neural network layer computations for the second context may be stored. The computation engine may then be configured as the third neural network layer to perform computations for the first context and the second context in parallel.

[0006]

[0013]

[0013] An embodiment of the present disclosure also relates to a method for scheduling operations in a computation engine for performing multi-layer neural network computations for multiple contexts. Referring to the above example, the scheduling method may include determining execution batches for computations of various neural network layers for various contexts. The scheduling may start with a highest-level neural network layer (e.g., a third neural network layer) and determine one or more execution batches for the highest-level neural network layer. One or more execution batches for lower layers (e.g., a second neural network layer) may then be determined and scheduled before the execution batch for the highest-level neural network layer. The determination of the execution batches may be based on available buffer space. If a buffer cannot provide the space required for computation of one neural network layer for all contexts, the computation may be divided into multiple batches, and the division may be performed so that the buffer can provide sufficient space to store output data generated from previous execution batches (to be provided to future execution batches) as well as input / output data for the currently executing batch.

[0007]

[0014] An artificial neural network (hereinafter referred to as a "neural network") may include multiple processing nodes. In the case of a deep neural network (DNN), the processing nodes may be divided into layers, including, for example, an input layer, several hidden layers (also known as hidden layers), and an output layer. Each processing node in the input layer receives an element of an input set and scales the element with a weight to indicate the element's influence on the output. The processing nodes in the hidden layer may combine the scaled elements received from each processing node in the input layer to calculate a set of intermediate outputs. For example, each processing node in the hidden layer may generate an intermediate output by calculating the sum of the products of the elements and weights and then applying an activation function to the sum. The intermediate output from each processing node in one hidden layer may be considered an activated vote (or no vote) associated with a weight indicating the influence of the vote to determine the intermediate output of the next hidden layer. The output layer may generate a sum of the scaled intermediate outputs from the final hidden layer and generate a binary output (e.g., "yes" or "no") based on whether the sum of the scaled intermediate outputs exceeds a threshold. Due to the combination of scaled elements between layers, the size of higher layers (e.g., output layer, intermediate layer immediately before the output layer, etc.) is usually smaller than the size of lower layers (e.g., input layer, intermediate layer immediately after the input layer, etc.).

[0008]

[0015] As described below, the processing node may be implemented with circuitry and data paths optimized for high-speed arithmetic operations, such as a graphics processing unit (GPU). For example, the circuitry may include an array of computational units linked together via a data path. The computational units may retrieve one or more weights and one or more inputs from a buffer and perform a scaling operation (e.g., scaling an input by a set of weights) and / or an addition operation (e.g., combining scaled inputs received from another computational unit). To perform multi-layer neural network processing, the data path may be configured to retrieve different sets of weights and inputs associated with different neural network layers from different buffers at different times and forward the weights and inputs to the computational units to perform the calculations. For example, in a first time period, the computational unit may be configured as an input layer, and the data path may be configured to retrieve a first set of weights and an input dataset from a buffer and forward the first set of weights and the input dataset to the computational unit. The computational units may perform the scaling and combination operations to calculate a first set of intermediate outputs and store the first set of intermediate outputs in a buffer separately from the weights and input dataset to avoid overwriting the weights and input dataset to be delivered to the computational units. In a second time period, the computation unit may be configured as a first hidden layer to compute a second set of intermediate outputs, and the data path may be configured to retrieve the first set of intermediate outputs and the second set of weights from the buffer and send the first set of intermediate outputs and the second set of weights to the computation unit to perform the computation. The process may be repeated until the computation unit performs the computation for the output layer.

[0009]

[0016] Such a configuration can reduce the number of computing units required to perform neural network processing, but the utilization of the computing units may decrease during processing. For example, as described above, the size of an upper layer is typically smaller than the size of a lower layer. If an array of computing units is configured to perform neural network processing on a single set of input data, the number of computing units involved in the calculation may decrease as data propagates from the lower layer to the higher layer. When the calculation for the upper layer is being performed, some of the computing units included in the calculation for the lower layer may be idle, leading to a decrease in utilization of the computing units.

[0010]

[0017] To achieve higher utilization, the array of computing units can be configured to process multiple input data sets (or input data associated with various contexts) in a lockstep manner. For example, the array of computing units can be configured to sequentially perform calculations for lower layers (e.g., input layer, lower hidden layer, etc.) for each context, where the calculations for the lower layers may include larger input data sets and weight sets and require more computing units and more buffer space. After the calculations for the lower layers are completed, the calculation results can be stored in a buffer. The array of computing units can then be configured to perform parallel calculations for multiple contexts of the next layer (e.g., upper hidden layer, output layer, etc.) based on the calculation results of the lower layers stored in the buffer. The calculations for the next layer may include smaller input data sets and weight sets and can be processed by a smaller number of computing units, and the buffer can also accommodate input and output data for the next layer calculations of a larger number of contexts.

[0011]

[0018] While such a configuration can maintain a relatively high utilization rate, a large buffer may be required to store the entire computation results for a particular layer of the neural network and provide intermediate outputs for the computation for the next layer. To further improve utilization for the computation of higher layers, the required size of the buffer may further increase when multiple data sets are input to the array of computation units. To implement the buffer, a larger number of memory units and associated interface circuits may be required. All of these may increase power consumption and chip space, which may significantly limit the adoption of computing devices and applications for calculations and problem solving using neural network architectures.

[0012]

[0019] Embodiments of the present disclosure relate to a method for operating a computation engine to perform computations for a multi-layer neural network for multiple contexts. In one example, a method includes: using a computational engine to perform a first computational task associated with a first neural network layer for a first context to generate first data; using the computational engine to perform, based on the first data, a second computational task associated with a second neural network layer for the first context to generate second data; storing the second data in a memory device; using the computational engine to perform a third computational task associated with the first neural network layer for the second context to generate third data; using the computational engine to perform, based on the third data, a fourth computational task associated with a second neural network layer for the second context to generate fourth data; storing the fourth data in the memory device; and using the computational engine to perform fifth and sixth computational tasks in parallel based on the second data and the fourth data stored in the memory device, the fifth and sixth computational tasks being associated with the third neural network layer for the first context and the second context, respectively.

[0013]

[0020]

[0010] Embodiments of the present disclosure also relate to a method for scheduling operations in a computation engine for performing multilayer neural network computation for multiple contexts. Referring to the above example, the scheduling method may include: determining a first execution batch including a fifth computation task and a sixth computation task; determining a second execution batch including the second computation task and a third execution batch including a fourth computation task based on a memory device having sufficient capacity to store input data and output data of the second computation task and output data of the fourth computation task, where both the second execution batch and the third execution batch are executed before the first execution batch; and determining a fourth execution batch including the first computation task to be executed between the second execution batch and the third execution batch based on a memory having sufficient capacity to store input data and output data of the first computation task and output data of the fourth computation task. The first, second, third, and fourth execution batches may be executed in the order determined above to perform the exemplary method for multilayer neural network computation for multiple contexts described above.

[0014]

[0021] Compared to the lock-step approach described above, embodiments of the present disclosure can operate with smaller buffers for storing intermediate computation outputs. As an illustrative example, in the lock-step approach, the computation engine can complete a first neural network layer computation for both a first context and a second context, then proceed to perform a second neural network layer computation for both the first context and the second context, followed by a third neural network layer computation for both the first context and the second context. The computation engine can perform a first neural network layer computation for the first context followed by a second context. To support batch processing, the buffer needs to provide a first storage space for storing input and output of the ongoing first neural network layer computation for the second context, as well as a second storage space for storing the output of the completed first neural network layer computation for the first context (to be provided as input to the second neural network layer computation for the first context after the first neural network layer computation for the second context is completed).

[0015]

[0022] On the other hand, in an embodiment of the present disclosure, the computation engine can complete the first neural network layer computation and the second neural network layer computation for the first context, store the second neural network layer output for the first context, and then proceed to perform the first neural network layer computation for the second context. To support the first neural network layer computation for the second context, the buffer may need to provide a first storage space (identical to the first storage space required in the illustrative example of the lock-step method described above) for the input data and output data of the first neural network layer computation for the second context. The buffer also needs to provide a second storage space to store the output of the completed second neural network layer computation for the first context. The second storage space in this case may be smaller than the second storage space in the example of the lock-step method described above. This is because the size of the second neural network layer is typically smaller than the size of the first neural network layer, and the output data size of the second neural network layer is also typically smaller than the output data size of the first neural network layer, so smaller buffers can be used to store intermediate results as the computation engine progresses through the layers.

[0016]

[0023] In embodiments of the present disclosure, computations on various input data sets can be arranged to maximize utilization of computational resources (e.g., by allowing simultaneous upper-tier computations on multiple input data sets) while reducing buffer usage. As a result, smaller memory devices can be used to support the computations, thereby reducing chip space and power consumption.

[0017]

[0024] 1 illustrates an exemplary classifier device 100 that processes a data sequence using the techniques disclosed herein. The classifier device 100 may be, for example, a computing device that operates a software application 102 and a predictive model 103 to predict information contained in a data sequence and perform a predetermined function based on the prediction. For example, the classifier device 100 may be part of an image recognition service that is provided to identify specific objects (e.g., text, people, etc.) from an image. The image recognition service is provided merely as an illustrative example, and it will be understood that the techniques disclosed herein can be used for other data processing applications including, for example, text-based data processing (e.g., processing search queries), audio data processing, etc.

[0018]

[0025] Image recognition services can be provided in a multi-tenant computing services system. A multi-tenant computing services system may typically include multiple servers capable of hosting data and may be used by multiple clients or organizations to run instances, such as virtual machine instances or bare metal instances (e.g., operating systems running directly on server hardware). Instances, such as bare metal or virtual machines, may be allocated to clients when they are needed by the multi-tenant computing services system and closed when they are no longer needed so that resources can be reallocated to other clients. In this disclosure, the terms “tenant,” “client,” and “customer” may be used interchangeably, although such terms do not necessarily imply the existence of any particular business arrangement. The term “instance” may refer, for example, to an instance running directly on server hardware or as a virtual machine. Different types of instances generally correspond to different hardware capabilities and / or hardware arrangements (e.g., different amounts of available memory and / or processing hardware). In the example of FIG. 1, the multi-tenant computing services system can provide image recognition services when a client needs them and shut them down when they are no longer needed so that the resources supporting the image recognition services (e.g., access to software application 102 and the underlying hardware resources for processing software application 102) can be reallocated to other clients.

[0019]

[0026] As shown in FIG. 1 , the software application 102 can receive pixel data of an image 104 from a user. The image 104 may include an array of pixels. The software application 102 can perform analysis on the pixel data to predict one or more objects 106 depicted in the image 104. The analysis may include, for example, comparing the pixel data to a set of predetermined image features. As described in more detail below, the software application 102 can utilize a predictive model 203 to calculate a set of scores based on the pixel data of the image 104. The set of scores may represent, for example, the likelihood that the image 104 contains the predetermined image features. The software application 102 can then determine other information about the content of the image 104 based on the scores. For example, based on the scores, the software application 102 can determine that the image 104 is an image of a panda.

[0020]

[0027] The predictive model 103 may be in the form of an artificial neural network. The artificial neural network may include multiple processing nodes, each configured to process a portion of the input pixel data or further process intermediate outputs from other processing nodes. Figure 2A shows an example of a predictive model 103 using the techniques disclosed herein. In the example of Figure 2A, the predictive model 103 may be a multi-layer neural network, such as a deep neural network (DNN), a convolutional neural network (CNN), or the like. The predictive model 103 may include an input layer 207, a set of hidden layers, including hidden layers 209 and 211, and an output layer (not shown in Figure 2A).

[0021]

[0028] Layer 207 may process pixel data representing various portions of image 104. In the example of FIG. 2A, layer 207 may process pixel data of image 104. Each processing node of layer 207 processes pixel values ​​(e.g., x0, x1, x2, ... x) corresponding to a given pixel in image 104. n) and are assigned to transmit one or more weights with the received pixel values ​​to layer 209. If prediction model 203 is a DNN, each processing node in layer 207 may be assigned a set of weights defined based on matrix W1. Each processing node in layer 207 may transmit the received pixel values ​​and assigned weights to each processing node in layer 209. If prediction model 103 is a CNN, a group of processing nodes in layer 207 may share a set of weights, and each group may transmit the set of weights and pixel values ​​received by the group of processing nodes to a single processing node in layer 209.

[0022]

[0029] Layer 209 may process the scaled outputs from layer 207 to generate a set of intermediate outputs. For example, assuming processing node 210a of layer 209 is connected to n processing nodes in layer 207, processing node 210a may generate a sum of the scaled outputs received from layer 207 based on the following equation:

[0023]

[0030] JPEG0007785848000001.jpg7113

[0024]

[0031] where Sum 210a represents the sum generated by processing node 210a. i ×x i represents the scaling of a particular pixel value (e.g., x0) with an associated weight (e.g., W10) by a processing node in layer 207. If the predictive model 203 is a DNN, each processing node in layer 209 generates a sum based on the scaling of pixel values ​​from each processing node in layer 207, and then calculates a sum (e.g., Sum 210a ) can be generated. The sum can also represent a dot product between an input vector containing several elements (e.g., pixel values) and a weight vector (e.g., W1).

[0025]

[0032] On the other hand, if the predictive model 103 is a CNN, each processing node in layer 209 may generate a sum based on scaling of pixel values ​​from a group of processing nodes in layer 207. The sum may represent the result of a convolution between a group of pixel values ​​and a filter containing weight values. FIG. 2B shows an example of a convolution operation that layer 209 may perform. In FIG. 2B, filter 230 may include a two-dimensional array of weights. The weights of filter 230 may represent the spatial distribution of pixels for some feature to be detected from the image. The two-dimensional array may have a height of R rows and a width of S columns, which is typically smaller than the input image, which has a height of H pixels and a width of W pixels. Each weight may map to a pixel within a rectangular block of pixel values ​​with the same R rows and S columns. A processing node of layer 209 (e.g., processing node 210a) may receive, from a group of processing nodes of input layer 207, group 240 of pixel values ​​corresponding to a first rectangular block of pixels from the input image and generate convolution output 242 based on a sum of multiplication results between each weight of filter 230 and each corresponding pixel in group 240 according to Equation 1 to generate an inner product between the matrix represented by filter 230 and the matrix represented by group 240. Another processing node of layer 209 may receive, from another group of processing nodes of input layer 207, group 244 of pixel values ​​corresponding to a second rectangular block of pixels from the input image and generate convolution output 246 based on a sum of multiplication results between each weight of filter 230 and each corresponding pixel in group 244 according to Equation 1 to generate an inner product between the matrix of filter 230 and the matrix represented by group 240. In some examples, each convolution output of FIG. 2B (e.g., convolution output 242, convolution output 246, etc.) may correspond to an output of a processing node of layer 209. In some examples, pixel data in an input image may be referred to as an input feature map to indicate that the pixels are processed by the same filter (or the same set of filters) that correspond to a particular feature, and the convolution output may be referred to as an output feature map to indicate that the output is the result of processing the input feature map with a filter.

[0026]

[0033] As shown in FIG. 2B , the convolution operations can be arranged within a sliding window such that the second rectangular block overlaps or is otherwise adjacent to the first rectangular block in the input image. For example, in the example of FIG. 2B , D may be the distance (in pixels) of the sliding window stride for each convolution operation, such that the block of pixels corresponding to group 244 may be located at a distance D (in pixels) from the block of pixels corresponding to group 240, and the next block of pixels may also be located at the same distance D from group 244. Other processing nodes in layer 209 may receive groups of pixels corresponding to other rectangular blocks and generate other intermediate outputs. The convolution outputs may be part of a convolution output array 280 having a height of E rows and a width of F columns. The array of convolution outputs may have a smaller height and width than the input image. The rectangular blocks of convolution outputs may be further grouped, and a convolution operation may be performed in layer 211 between the group of convolution outputs and another set of filter weights to generate another set of convolution outputs.

[0027]

[0034] In some examples, a convolution operation can be performed between multiple images and multiple filters. For example, with reference to FIG. 2C , a set of C filters 260 can correspond to a number (C) of images 270, and a convolution operation can be performed between each filter in the set of filters 260 and a block of pixels on a corresponding one of images 270. The convolution results for each filter-image pair can be summed to generate a convolution output as follows:

[0028]

[0035] JPEG0007785848000002.jpg8156

[0029]

[0036] Here, the convolution operation is performed on an image (or pixel array) X C eD+r,fD+swhich may refer to the value of a pixel at an image with index c in a number (C) image 370 with horizontal pixel coordinate of eD+r and vertical pixel coordinate of fD+s. D is the stride distance of the sliding window, while e and f correspond to the position of the output in the convolution output array, which may correspond to a particular sliding window. Furthermore, r and s correspond to a particular position within the sliding window. A pixel at location (r,s) and an image with index c is represented by a weight W in the corresponding filter of the same index c at the same (r,s) location. c r,s Equation 2 can also be used to express the convolution output 0 e,f To compute c r,s It is shown that the images may be multiplied by . A partial sum of the multiplication products within each sliding window for each of the images in the image set may be calculated, and then a sum of the partial sums for all images in the image set may be calculated.

[0030]

[0037] Moreover, in some examples, multiple sets of filters can be used to perform a convolution operation on a set of images to generate a set of convolved output arrays, each convolved output array corresponding to a set of filters. For example, the multiple sets of filters can correspond to multiple image features to be detected from the set of images, each convolved output array corresponding to the detection result for each image feature from the set of images. For example, if M sets of filters are applied to C images to generate M convolved output arrays, Equation 2 can be updated as follows:

[0031]

[0038] JPEG0007785848000003.jpg8165

[0032]

[0039] Here, the convolution output is 0 e,f m and weight Wc,m r,s has an index m corresponding to one of the M sets of filters.

[0033]

[0040] FIG. 2D shows an example of C sets of input data sets (C=3) to be convolved with M sets of filters (M=2). Each set of input data corresponds to an entry in a pixel group. For example, pixel groups 282, 286, and 290 may each correspond to one input data set. Each of the M sets of filters includes a set of C filters corresponding to the C sets of input pixel arrays. In the example of FIG. 2D, there are two filter sets: the first filter set includes filter arrays 284a, 284b, and 284c, and the second filter set includes filter arrays 288a, 288b, and 288c. The convolution operation generates M sets of output data sets, each corresponding to a convolution output array. In the example of FIG. 2D, two convolution output arrays 294 and 296 are generated. Each convolution output array corresponds to convolving one set (of the M sets) of filters with the input pixel array. For example, the first element O of the convolution output array 294 0,0 0 can be generated by the sum of the dot products between pixel group 282 and filter array 284a, the dot products between pixel group 286 and filter array 284b, and the dot products between pixel group 290 and filter array 284c.

[0034]

[0041] Referring again to FIG. 2A , one processing node in layer 209 may be configured to generate one convolution output array, and the set M of processing nodes in layer 209 may correspond to the set M of convolution output arrays. The processing nodes in layer 209 may also post-process each convolution output with, for example, an activation function to generate a final output for layer 209. The activation function may convert a convolution output (or a subsample) into a decision of whether to forward the convolution output (or subsample) to an upper layer. The generation of the decision may be similar to the firing of a biological neuron. One example of an activation function may be the rectified linear unit (ReLu), defined according to the following equation:

[0035]

[0042] ReLu(y)=max(0,y) (Equation 4)

[0036]

[0043] Processing nodes in layer 209 (e.g., processing node 210a) may process the convolution output subsamples with a ReLu function to generate intermediate outputs based on Equation 4. In addition, processing nodes in layer 209 may perform other post-processing operations, such as pooling operations, to reduce the number of intermediate outputs. Layer 209 may provide the post-processed intermediate outputs to layer 211, which may perform additional convolution operations based on different sets of filters. Outputs from each processing node in layer 211 may be forwarded to other higher intermediate layers or to an output layer (not shown in FIG. 2A). The output layer may, for example, form an output vector representing the probability that a particular image feature is included in image 104 in FIG. 2A (or a set of images represented by a set of input pixel arrays in FIG. 2D ) to determine whether an image or set of images associated with a particular context includes an image of a panda. For example, the output vector may be compared with a reference vector associated with a panda nose object or a reference vector associated with a panda, and a decision may be made based on the comparison result. Multiple instances of model 103 can be used to process multiple images (or multiple sets of images) associated with multiple contexts to produce a decision for each of the contexts.

[0037]

[0044] FIG. 3A illustrates an apparatus 300 according to some embodiments of the present disclosure. The apparatus 300 may be part of a computer system, e.g., a host server. The apparatus 300 may be part of a multi-tenant computational service system and may communicate with a host device (not shown in FIG. 3A ) to provide computational and memory resources to the computational service. For example, referring again to FIG. 1 , the apparatus 300 may provide computational and memory resources for computations using the predictive model 103. The host device may run a software application 102 and communicate with the apparatus 300 to perform one or more image recognition tasks based on the computations using the predictive model 103. The host device may transmit multiple image datasets associated with multiple contexts and provide the multiple image datasets to the apparatus 300, which may generate multiple outputs, e.g., predict whether each of the multiple image datasets includes a predetermined object.

[0038]

[0045] In the example of FIG. 3A, the apparatus 300 may include a neural network processor 302, a direct memory access (DMA) controller 316, and a host interface 314 coupled to a memory 312 via an interconnect 318. As described in more detail, the neural network processor 302 can provide computational resources to support computations using the predictive model 103. The memory 312 may be configured to store instructions received from a host device, input data (e.g., pixel groups 282, 286, and 290 of FIG. 2D), and weights (e.g., filter arrays 284a, 284b, 284c, 288a, 288b, and 288c). The memory 312 may also be configured to store outputs of the neural network processor 302 (e.g., convolution output arrays 294 and 296 of FIG. 2D). The memory 312 may include any suitable memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate DRAM (DDR DRAM), storage class memory (SCM), flat memory devices, etc.

[0039]

[0046] DMA controller 316 may be configured to perform DMA operations to transfer data between neural network processor 302 and a host device. For example, as described above, the host device may store instructions, input data, and weights in memory 312. The host device may provide memory addresses of the stored instructions, data, and weights to neural network processor 302 (e.g., in the form of memory descriptors). Neural network processor 302 may then retrieve the stored instructions, data, and weights based on the memory addresses provided by the host device. Neural network processor 302 may also store computation results (e.g., one or more image recognition decisions) in memory 312 and provide the memory addresses of the stored results to the host device.

[0040]

[0047] Host interface 314 may be configured to enable communication between a host device and neural network processor 302. For example, host interface 314 may be configured to transmit memory descriptors including memory addresses of stored data (e.g., input data, weights, computation results, etc.) between the host device and neural network processor 302. Host interface 314 may include, for example, a Peripheral Component Interconnect Express (PCIe) interface or any suitable interface for communicating with a host device.

[0041]

[0048] Neural network processor 302 can provide computational resources to support computations with one or more instances of predictive model 103. In the example of FIG. 3A, neural network processor 302 can include a state buffer 322, a computation engine 324, an output buffer 328, a post-processor 329, and a computation controller 330.

[0042]

[0049] The state buffer 322 may be configured to provide caching of data used in the computation in the computation engine 324. The data cached in the state buffer 322 may include, for example, input data and weights retrieved from the memory 312, as well as intermediate outputs of the computation in the computation engine 324. Caching can reduce the impact of memory access bottlenecks (e.g., caused by latencies in the memory 312, the DMA controller 316, the interconnect 318, etc.) on the performance of the computation engine 324. The state buffer 322 can be controlled by the computation controller 330 to pre-fetch a set of weights to the computation engine 324 and then fetch the input data as a sequential stream to the computation engine 324, which performs the computation and generates a sequential stream of intermediate output data. The stream of intermediate output data can be collected in the output buffer 328 and post-processed by the post-processor 329. The post-processed stream of intermediate output data can be stored in the state buffer 322. The intermediate output data may be stored in a location in the state buffer 322 separate from the locations where the input data and weights are stored, e.g., to avoid overwriting input data that has not yet been fetched to the computation engine 324. The state buffer 322 may be an on-chip memory device and may include, for example, static random access memory (SRAM).

[0043]

[0050] The computation engine 324 may include a set of circuits configured to perform one or more arithmetic operations involved in the neural network computation. For example, the computation engine 324 may include a set of multipliers that perform scaling of input data with associated weights, as described above, and a set of adders that generate a sum of multiplication results that represent the dot product and convolution result. The computation engine 324 may obtain the weights used for scaling from the state buffer 322. As described in more detail below, the computation engine 324 may obtain the input to be scaled from the state buffer 322 or from the output buffer 328.

[0044]

[0051] In some examples, the computation engine 324 may be controlled (e.g., by the computation controller 330) to sequentially perform computations for various neural network layers to reduce the size of the weights stored in the state buffer 322. For example, the computation engine 324 may be controlled to perform arithmetic operations for one neural network layer (e.g., layer 207) within one time period, and then perform arithmetic operations for the next neural network layer (e.g., layer 209) within the next time period. In such a configuration, the state buffer 322 may pre-fetch and cache a set of weights for one neural network layer instead of caching multiple sets of weights for multiple neural network layers to support the arithmetic operations in the computation engine 324. As a result, the size required for the state buffer 322 may be reduced, and the chip size and power consumption of the neural network processor 302 may also be reduced. On the other hand, as described above, the state buffer 322 needs to provide storage space for both input data (and weights) and output data for the computation of at least one neural network layer computation for one context (e.g., an image, a set of images, audio data, or other data associated with the context) to avoid overwriting input data with output data for the computation.

[0045]

[0052] 3B shows an example of the internal structure of the computation engine 324. The computation engine 324 may include an array of processing elements (PEs) arranged in multiple rows and columns. The PEs in a row are connected to an internal row bus, XBUS INTERNAL , but each of the PEs 11, 21, 31, ... x1 is connected to an external row bus XBUS EXTERNAL Moreover, the PEs in one column are connected to the external circuit via the internal column bus YBUS INTERNAL , and each of the PEs 11, 12, 13, ... 1y is connected to an external column bus YBUS EXTERNALAs will be explained in more detail below, the external row bus XBUS EXTERNAL and external column bus YBUS EXTERNAL can be configured to provide data from, for example, the state buffer 322 and the output buffer 328 to the PE array in order to scale the computational operations for a particular neural network layer.

[0046]

[0053] In some configurations, each column of PEs may perform the scaling and summing calculations for one processing node of predictive model 103 (e.g., one of processing nodes 208a, 210a, 212a, etc.), while each row of PEs may perform the scaling and summing calculations for one input data set. As an illustrative example, to perform the calculations for layer 209, each of PEs 11, 12, 13, ... 1y in the first row may connect to an XBUS EXTERNAL 2D. Each of PEs 11, 21, 31, ..., x1 in the first column can receive elements of filter sets 284a, 284b, and 284c to generate the output of convolution output array 294 in FIG. 2D. Additionally, each of PEs 21, 22, 23, ..., 2y in the second row can receive the pixel values ​​of pixel group 282 in FIG. 2D from XBUS 11. EXTERNAL2D from the first column, while PEs 12, 22, 32, ..., x2 in the second column can each receive elements of filter sets 288a, 288b, and 288c to generate the output of convolution output array 296 in FIG. 2D. Each PE includes a multiplier that performs a multiplication function between the received pixel value and the received weight to generate a multiplication result. Additionally, each PE also includes an adder that accumulates multiplication results within one column of PEs. For example, PE 11 generates a first multiplication result based on a first pixel value and a first weight and passes the first multiplication result to PE 21 via the YBUS. PE 21 generates a second multiplication result based on a second pixel value and a second weight, adds the second multiplication result to generate a partial sum, and passes the partial sum to PE 31 via another YBUS. Other PEs in the same column as PEs 11, 21, and 31 also perform multiplication and accumulation. PEx1 then generates the convolution output 242 and sends it to YBUS EXTERNAL The convolution output can be sent via

[0047]

[0054] In some configurations, PEs in the computation engine 324 can be used to implement multiple instances of the predictive model 103. For example, when processing higher-level layers with a small number of processing nodes and a small number of input data, PEs in the same row can be used to process input datasets of different contexts (corresponding to different images or sets of images captured by different people at different locations, times, or otherwise conveying different information). For example, PE 11 may be configured to process one input dataset of a first context, while PE 12 may be configured to process another input dataset of a second context. Both PE 11 and PE 12 can store and apply the same set of weights to the input data. A first row (including PEs 11, 21, ... x1) may be configured to process multiple input datasets of the first context, while a second row (including PEs 12, 22, ... x2) may be configured to process multiple input datasets of the second context.

[0048]

[0055] 3C shows an example of the internal structure of a PE 340, which may be any one of the PEs described in FIG. 3B. The PE 340 may include a splitter 350, a multiplier 353, and an adder 356. The splitter 350 receives an input XBUS (e.g., an XBUS INPUT ) and extracts input values ​​and weights. For example, PE11 can receive a set of data from XBUS IN PE11 can receive input values ​​and weight sets for not only PE11 but also other PEs in the same row from the XBUS. IN The adder 356 can extract the input value (e.g., IN 351) and weight value (e.g., W 352) for the PE 11 from the above data and provide the extracted value to the multiplier 353 to generate the multiplication result (e.g., M 354). INPUT ) from the previous PE in the same column to the multiplication result (e.g., M354) to generate the partial sum, and output it to the output YBUS (e.g., YBUS OUTPUT ) to the next PE in the same column. The PE can send the partial sum to the next PE in the same column via the output XBUS (XBUS OUTPUT ) to the next PE in the same row via XBUS IN You can also transfer data.

[0049]

[0056] 3A , the neural network processor 302 further includes an output buffer 328 and a post-processor 329. The output buffer 328 includes a set of registers that store the output of the computation engine 324. The post-processor 329 may include circuitry (e.g., a multiplexer circuit configured as a mapping table) that applies one or more activation functions (e.g., a ReLu function) to the output of the computation engine 324. The post-processor 329 may also include circuitry that performs a pooling operation to reduce the size of the output. The post-processor 329 can store the post-processed output in the state buffer 322.

[0050]

[0057] The calculation controller 330 is an XBUS EXTERNAL and YBUS EXTERNAL , and control the computations in the computation engine 324. For example, the computation controller 330 may control the computation engine 324 to sequentially perform the computations for each neural network layer by providing the input data and weights associated with each neural network layer at various times. As an illustrative example, as a first step, the computation controller 330 obtains the input pixel data and weights associated with the scaling operation in layer 209 and transmits them to the computation engine 324 via the XBUS. EXTERNAL Input pixel data and weights can be provided to the computation engine 324 via the YBUS. The computation engine 324 then generates the output and EXTERNAL to provide a set of intermediate outputs (e.g., first_intermediate_output 210a ) may be generated. The intermediate output may be stored in an output buffer 328. If the subsequent calculation is for the same input data set in the next layer (e.g., layer 211), calculation controller 330 retrieves the intermediate outputs stored in output buffer 328 and transfers them to the XBUS bus from state buffer 322 to begin the calculation in layer 211. EXTERNAL The intermediate outputs, as well as a new set of weights, can be provided to the computation engine 324 via the computation controller 330. On the other hand, if the intermediate outputs will not be used in subsequent computations, the computation controller 330 can store the intermediate outputs in the state buffer 322. The process can be repeated for each layer until the computations at the output layer are complete. The order of computations can be determined by a scheduler module 332 that operates a predetermined scheduling scheme described below.

[0051]

[0058] As described above, in DNNs and CNNs, the size of higher-level hidden layers is typically smaller than the size of lower-level hidden layers. Therefore, as neural network calculations progress through the neural network layers toward the output layer, the computational resources used for the calculations for a layer may decrease over time. Referring to the example of FIG. 3B , layer 209 may include the same number of processing nodes as the number of PE columns in computation engine 324. Thus, when performing calculations for layer 209, every column in computation engine 324 participates in the calculation, and each column of PEs generates an output corresponding to one processing node in layer 209. However, if layer 211 has only half of the processing nodes in layer 209, half of the PE columns are not required for the calculation and may be idle, leading to insufficient utilization of computation engine 324. To improve utilization of computation engine 324, computation controller 330 can control computation engine 324 to process multiple input data sets (e.g., multiple images or other contexts) and generate multiple outputs for the multiple input data sets, as described above.

[0052]

[0059] Referring now to Figure 4, Figure 4 shows an example sequence 400 for operating the computation engine 324 to process multiple input data sets. The blocks on the left show a sequence of computations in the computation engine 324, while the blocks on the right show a corresponding sequence of intermediate outputs stored in the state buffer 322. The storage of weights in the state buffer 322 is not shown in Figure 4.

[0053]

[0060] In some examples, the sequence 400 can be executed to operate the computation engines 324 in a lock-step manner. For example, computations for lower-level neural network layers can be performed sequentially for each input data set (when computational resource requirements are substantial and the computation engines 324 are fully utilized). After the computations for one neural network layer for all input data sets (for various contexts) are completed, the computation engines 324 then perform the computations for the next neural network layer. As the computations reach higher-level neural network layers with reduced computational resource requirements, the computation controller 330 can perform parallel computations on the input data sets to maintain utilization of the computation engines 324. As described herein and depicted in the figures, "C-1" may refer to the first context, "C-2" may refer to the second context, "C-3" may refer to the third context, and "C-4" may refer to the fourth context.

[0054]

[0061] At T0, the computation controller 330 can control the computation engine 324 to perform the L0 neural network layer (e.g., layer 209) computation for C-1, thereby fully utilizing the computation engine 324. At T0, the state buffer 322 needs to provide storage space to store the input data (and weights) for the C-1 L0 computation and the output data of the C-1 L0 computation.

[0055]

[0062] At T1, after the L0 neural network layer calculation for C-1 is completed, the calculation controller 330 can store the output data of the L0 calculation for C-1 in the state buffer 322 and then control the calculation engine 324 to perform the L0 neural network layer calculation for C-2. At T1, the state buffer 322 needs to provide storage space to store the input and output data of the ongoing L0 calculation for C-2 and the output data of the completed L0 calculation for C-1.

[0056]

[0063] At T2, the computation controller 330 can control the computation engine 324 to perform the L0 neural network layer computation for C-3. At T2, the state buffer 322 needs to provide storage space to store the input and output data of the ongoing L0 computation for C-3 and the output data of the completed L0 computation for C-1 and C-2.

[0057]

[0064] At T3, the computation controller 330 can control the computation engine 324 to perform the L0 neural network layer computation for C-4. At T3, the state buffer 322 needs to provide storage space to store the input and output data of the ongoing L0 computation for C-4 and the output data of the completed L0 computations for C-1, C-2, and C-3.

[0058]

[0065] At T4, the computation controller 330 can switch the computation engine 324 to perform an L1 neural network layer computation (e.g., layer 211). The computation controller 330 can control the computation engine 324 to perform L1 computations for C-1 and C-2 in parallel. At T4, the state buffer 322 needs to provide storage space to store output data of the ongoing L1 computations for C-1 and C-2, and output data of the completed L0 computations for C-1, C-2, C-3, and C-4. The total size of the data stored at T4 may be the largest compared to other stages of the sequence 400.

[0059]

[0066] At T5, the computation controller 330 can control the computation engine 324 to perform the L1 computation for C-3 and C-4. At T5, the state buffer 322 needs to provide storage space to store the output data of the ongoing L1 computation for C-3 and C-4, as well as the output data of the completed L0 computation for C-3 and C-4. Because the L1 computation for C-1 and C-2 completes at T5 and their output data is not needed, the output data of the completed L0 computation for C-1 and C-2 can be deleted (or overwritten by the output data of the L1 computation for C-3 and C-4).

[0060]

[0067] At T6, the computation controller 330 can switch the computation engines 324 to perform L2 neural network layer computations (e.g., output layers). The computation controller 330 can control the computation engines 324 to perform L2 computations for C-1, C-2, C-3, and C-4 in parallel. At T6, the state buffer 322 needs to provide storage space to store output data of the ongoing L2 computations for C-1, C-2, C-3, and C-4, as well as input data for those computations (e.g., output data of the L1 computations for C-1, C-2, C-3, and C-4).

[0061]

[0068] 4 allows the computation engine 324 to be kept at full capacity, but requires a large state buffer 322 to store the outputs of the low-level neural network (e.g., L0) computations for multiple contexts, such as time T4. These intermediate outputs are typically large in size and require a significant amount of on-chip memory space for storage.

[0062]

[0069] Referring now to Figure 5, Figure 5 illustrates an example sequence 500 for operating the computation engines 324 to process multiple input data sets. The sequence 500 can be implemented to operate the computation engines 324 in a non-lockstep manner. The blocks on the left illustrate a sequence of computations in the computation engines 324, while the blocks on the right illustrate a corresponding sequence of intermediate outputs stored in the state buffer 322. The storage of weights in the state buffer 322 is not shown in Figure 5.

[0063]

[0070] At T0, the computation controller 330 can control the computation engine 324 to perform the L0 neural network layer (e.g., layer 209) computation for C-1, thereby fully utilizing the computation engine 324. At T0, the state buffer 322 needs to provide storage space to store the input data (and weights) for the C-1 L0 computation and the output data of the C-1 L0 computation.

[0064]

[0071] At T1, after the L0 neural network layer calculation for C-1 is completed, the calculation controller 330 can store the output data of the L0 calculation for C-1 in the state buffer 322 and then control the calculation engine 324 to perform the L0 neural network layer calculation for C-2. At T1, the state buffer 322 needs to provide storage space to store the input and output data of the ongoing L0 calculation for C-2 and the output data of the completed L0 calculation for C-1.

[0065]

[0072] At T2, the computation controller 330 can switch the computation engine 324 to perform the L1 neural network layer computations (e.g., layer 211) for C-1 and C-2. At T2, the state buffer 322 needs to provide storage space to store input data for the ongoing L1 computations (e.g., output data of the L0 computations for C-1 and C-2), as well as output data generated by the L1 computations.

[0066]

[0073] At T3, the computation controller 330 can once again switch the computation engine 324 to perform the L0 neural network layer computation for C-3. At T3, the state buffer 322 needs to provide storage space to store the output data of the completed L1 computations for C-1 and C-2, as well as the input and output data for the ongoing L0 computation for C-3.

[0067]

[0074] At T4, the computation controller 330 can control the computation engine 324 to perform an L0 neural network layer computation for C-4. At T4, the state buffer 322 needs to provide storage space to store output data of the completed L1 computation for C-1 and C-2, output data of the completed L0 computation for C-3, and input and output data for the ongoing L0 computation for C-4. Compared to T4 in sequence 400, storing L0 output data for fewer contexts (which is typically larger than the upper neural network layers) can reduce the required size of the state buffer 322.

[0068]

[0075] At T5, the computation controller 330 can control the computation engines 324 to perform the L1 neural network computations for C-3 and C-4 in parallel. At T5, the state buffer 322 needs to provide storage space to store the output data of the completed L1 computations for C-1 and C-2, as well as the input and output data for the ongoing L1 computations for C-3 and C-4. The input data for the ongoing L1 computations for C-3 and C-4 includes the output data of the completed L0 computations for C-3 and C-4.

[0069]

[0076] At T6, the computation controller 330 can switch the computation engines 324 to perform L2 neural network layer computations (e.g., output layers). The computation controller 330 can control the computation engines 324 to perform L2 computations for C-1, C-2, C-3, and C-4 in parallel. At T6, the state buffer 322 needs to provide storage space to store output data of the ongoing L2 computations for C-1, C-2, C-3, and C-4, as well as input data for those computations (e.g., output data of the L1 computations for C-1, C-2, C-3, and C-4).

[0070]

[0077] 6 shows an example flow diagram of a process 600 for processing multiple data sets associated with various contexts in a multi-layer neural network according to some aspects of the present disclosure. The multi-layer neural network may include a first neural network layer (e.g., layer 209), a second neural network layer (e.g., layer 211) that depends on the first neural network layer, and a third neural network layer (e.g., output layer) that depends on the second neural network layer. The process may be implemented by various systems described above, such as, for example, the computational controller 330 and the computational engine 324.

[0071]

[0078] In operation 602, a computational engine (e.g., computational engine 324) may be configured as a first neural network layer to process a first dataset (e.g., of a first context) and generate a first layer output for the first dataset.

[0072]

[0079] At operation 604, the computational engine, while configured as a first neural network layer, may process a second data set (e.g., of a second context) to generate a first layer output for the second data set.

[0073]

[0080] In operation 606, the computational engine may be configured as a second neural network layer and process the first layer output of the first data set and the first layer output of the second data set to generate a second layer output of the first data set (for the first context) and a second layer output of the second data set (for the second context), respectively.

[0074]

[0081] At operation 608, the computational engine is configured to return to the first neural network layer after generating the second layer output for the first dataset and the second layer output for the second dataset at operation 606, and the computational engine can process a third dataset (e.g., for a third context) to generate a first layer output for the third dataset.

[0075]

[0082] At operation 610, the computational engine, while configured as a first neural network layer, may process a fourth data set (e.g., for a fourth context) to generate a first layer output for the fourth data set.

[0076]

[0083] In operation 612, the computational engine may be configured as a second neural network and may process the first layer output of the third data set (for the third context) and the first layer output of the fourth data set (for the fourth context) to generate a second layer output of the third data set and a second layer output of the fourth data set, respectively.

[0077]

[0084] In operation 614, the computational engine may be configured as a third neural network and may process the second layer output of the first dataset (of the first context), the second layer output of the second dataset (of the second context), the second layer output of the third dataset (of the third context), and the second layer output of the fourth dataset (of the fourth context) to generate a third layer output of the first dataset, a third layer output of the second dataset, a third layer output of the third dataset, and a third layer output of the fourth dataset, respectively.

[0078]

[0085] In some examples, rearranging the order of computation from a lockstep sequence (e.g., of FIG. 4) to a non-lockstep sequence (e.g., of FIG. 5) may be based on a scheduling scheme. The scheduling scheme may be used to determine a sequence of execution batches, each including one or more computation tasks for a neural network layer to execute in parallel in the computation engine 324. Each computation task may be executed for a context, and each of the execution batches may be executed at different times. The objective of the scheduling scheme may be to maximize each execution batch size (e.g., the number of parallel computation tasks executed) subject to the constraints of available space in the state buffer 322 and inter-layer data dependencies.

[0079]

[0086] In some examples, the scheduler module 332 of the computation controller 330 can apply a graph reduction scheme to determine a graph of dependencies between computations of various neural network layers for various contexts. The scheduler module 332 can then traverse the graph, starting with computations of higher neural network layers, to determine the sequence of computation tasks. The determination can be based on the number of parallel computation tasks that the state buffer can support and inter-layer data dependencies. As described above, to support parallel execution of several computation tasks of a neural network layer, the state buffer needs to have sufficient capacity to store both input data and output data for several computation tasks. After determining the first execution batch size for a first neural network layer (e.g., layer 211 of FIG. 2A), the scheduler module 332 can traverse the dependency graph and determine whether the state buffer can support several computation tasks for a second neural network layer (e.g., layer 209 of FIG. 2A), which provides input data to the first neural network layer. If the state buffer cannot support some of the computational tasks for the second neural network layer, the scheduler can divide the computational tasks for the second neural network layer into multiple execution batches. The determination of the multiple execution batches can be to maximize the number of parallel computational tasks for the second neural network layer in each execution batch, subject to the constraint that the state buffer has sufficient capacity to store the output data of previous execution batches as well as the input / output data of the ongoing execution batch. After determining the execution batches for the second neural network layer, the scheduler can traverse the dependency graph to determine computational tasks for a third neural network (e.g., layer 207 in FIG. 2A) that provides input data for the second neural network layer, and repeat the execution batches until a determination operation is made, subject to the state buffer capacity constraint described above.

[0080]

[0087] 7A-7G, which illustrate an example scheduling scheme. FIG. 7A illustrates an example neural network layer computation dependency graph for contexts C-1, C-2, C-3, and C-4 of FIGS. 4 and 5. In this example, it can be assumed that computation engine 324 imposes no constraints on the number of computational tasks that can be executed simultaneously; the only constraints arise from inter-layer dependencies and state buffer capacity. In this illustrative example, the state buffer size may be 2 units.

[0081]

[0088] In the example of FIG. 7A, each bubble (i.e., circle) can correspond to a computation task or external input for a context of a neural network layer, and the number in the bubble indicates the size of the output data to be generated by the computation task or the size of the external input data. The size of the output data (or external input data) is related to the size of the state buffer, both in the same units. Each execution batch can include one or more bubbles of the same neural network layer. Pairs of bubbles are linked by respective arrows whose direction indicates data dependency. The number adjacent to the arrow can indicate the size of the weight value used for the computation task of the neural network layer. The weight value can be shared between bubbles within one execution batch. Referring to the dependency graph of context C-1, the L0 (neural network layer 0) computation task can receive image data of 0.5 units in size as an external input and weights of 0.3 units in size as input data. The L0 computation task can generate output data of 0.3 units in size. The L1 (neural network layer 1) computation task can receive as input data the output data of the L0 computation task (which is 0.3 units in size) and weights of 0.25 units in size. Furthermore, the L2 (neural network layer 2) computation task can receive as input data the output data of the L1 computation task (which is 0.2 units in size) and weights of 0.1 units in size. Furthermore, the L2 computation task can generate output data of 0.1 units in size.

[0082]

[0089] 7B shows a scheduling operation 702. In operation 702, the scheduler determines a first batch size for L2 computation. To support concurrent L2 computation for C-1, C-2, C-3, and C-4, the state buffer needs to have the capacity to store the input and output data of the concurrent L2 computation. The input data includes L1 output data for four contexts with a total size of 0.8 units (0.2 × 4) and a shared weight of 0.1 units, while the output data size is 0.4 units (0.1 × 4). The required capacity is 1.3 units (0.8 + 0.4 + 0.1), which is less than the capacity of the state buffer (2 units). Based on this, the scheduler can include L2 computation tasks for all four contexts in the first execution batch to be executed at time T1.

[0083]

[0090] 7C and 7D show scheduling operation 704. In operation 704, the scheduler determines whether the L1 computations of the four contexts can be executed within the second execution batch. The scheduler can determine whether the state buffer has the capacity to store the input and output data of the concurrent L1 computations for all four contexts. Referring to FIG. 7C, the input data includes L0 output data for four contexts with a total size of 1.2 units (0.3 × 4) and a shared weight size of 0.25 units, while the output data size is 0.8 units (0.2 × 4). The required capacity is 2.25 units (1.2 + 0.8 + 0.25), which exceeds the capacity of the state buffer. Based on this determination, the scheduler can divide the L1 computations into two or more execution batches. For example, the scheduler may determine the number of L1 computational tasks that can be included in an execution batch based on the state buffer having sufficient capacity to store input data and output data of several L1 computational tasks in that execution batch, as well as output data of L1 computational tasks of previous execution batches.

[0084]

[0091] As an illustrative example, referring to FIG. 7D , the scheduler may determine a second execution batch including L1 computation tasks for C-2 and C-3, and a third execution batch including L1 computation tasks for C-1 and C-2. The third execution batch should be executed before the second execution batch. The total input / output data size of the second execution batch may be 1.25 units (0.2×2 + 0.3×2 + 0.25), while the output data size of the third execution batch is 0.4 units (0.2×2). Based on the total data size (1.65 units) being smaller than the capacity of the state buffer, the scheduler schedules the second execution batch at time T2 and the third execution batch at time T3.

[0085]

[0092] 7E and 7F show scheduling operation 706. In operation 706, the scheduler determines whether the L0 computations for C-2 and C-3 can be executed in parallel as a single batch (to provide input data for the second execution batch). Referring to FIG. 7E, to support the parallel L0 computations for C-2 and C-3 to be executed between times T1 and T2 (to avoid the state buffer needing to store the output of the L0 computations for C-1, C-2, C-3, and C-4), the state buffer needs to store the input / output data of the parallel L0 computations for C-2 and C-3, as well as the output data of the third execution batch. The total input / output data size of the L0 computations for C-2 and C-3 is 1.9 units (0.5 × 2 + 0.3 × 2 + 0.3), while the output data size of the third execution batch is 0.4 units. Based on the total data size (2.3 units) exceeding the capacity of the state buffer, the scheduler determines to split the L0 computations into two batches. 7F, the scheduler may determine a fourth execution batch including an L0 calculation for C-4 and a fifth execution batch including an L0 calculation for C-3. The scheduler may also determine that the state buffer has sufficient capacity to store the input and output data for the fourth execution batch (having a data size of 1.3 units), the output data for the fifth execution batch (having a data size of 0.3 units), and the output data for the third execution batch (having a data size of 0.4 units), all of which have a total data size of 2 units. The scheduler may then schedule the fourth execution batch at time T4 and the fifth execution batch at time T5.

[0086]

[0093] FIG. 7G illustrates scheduling operation 708. In operation 708, the scheduler determines whether the L0 computations for C-1 and C-2 can be executed in parallel as a single batch (to provide input data for the third execution batch). Referring to FIG. 7G, to support the parallel L0 computations for C-1 and C-2 to be executed before the execution batch determined in operations 702-706, the state buffer needs to store input / output data for the parallel L0 computations for C-1 and C-2. The total input / output data size of the L0 computations for C-1 and C-2 is 1.9 units (0.5×2+0.3×2+0.3), which is less than the capacity of the state buffer. The scheduler can then include the L0 computations for C-1 and C-2 in the sixth execution batch and schedule the sixth execution batch at T6.

[0087]

[0094] 8 shows an example flow diagram of a process 800 for scheduling operations in a computation engine (e.g., computation engine 324) coupled with a memory device (e.g., state buffer 322). The process may be implemented by various systems described above, such as, for example, scheduler module 332 of neural network processor 302.

[0088]

[0095] In operation 802, the scheduler module may determine a set of computational tasks to be executed, the set of computational tasks including a first computational task, a second computational task, a third computational task, and a fourth computational task, wherein the third computational task and the fourth computational task generate input data for the first computational task and input data for the second computational task, respectively.

[0089]

[0096] In operation 804, the scheduling module may determine a first execution batch including the first computational task and the second computational task based on the state buffer having sufficient capacity to hold input / output data of both the first computational task and the second computational task.

[0090]

[0097] At operation 806, the scheduling module may determine whether the memory device has sufficient capacity to hold input data and output data for both the third and fourth computational tasks. If the memory device has sufficient capacity to hold input data and output data for both the third and fourth computational tasks (at operation 808), the scheduling module may, at operation 810, determine a second execution batch including the third and fourth computational tasks and schedule the second execution batch to execute before the first execution batch.

[0091]

[0098] On the other hand, if the memory device does not have sufficient capacity to hold the input data and output data of both the third and fourth computational tasks (at Operation 808), but does have sufficient capacity to hold the output data of the third computational task and the input data and output data of the fourth computational task (at Operation 812), the scheduling module may determine a third execution batch that includes the third computational task at Operation 814. The scheduling module may also determine a fourth execution batch that includes the fourth computational task at Operation 816. The scheduling module may schedule the third execution batch to be executed first, followed by the fourth execution batch, followed by the first execution batch.

[0092]

[0099] In the example process 800, the first and second computational tasks may be second neural network layer computational tasks for the first and second contexts, respectively. Additionally, the third and fourth computational tasks may be first neural network layer computational tasks for the first and second contexts, respectively.

[0093]

[0100] 9 illustrates an example of a computing device 900. The functionality and / or certain components of computing device 900 may be used without limitation with other embodiments disclosed elsewhere in this disclosure. Computing device 900 may perform computations to facilitate processing of tasks. As an illustrative example, computing device 900 may be part of a server in a multi-tenant computing services system. Various hardware and software resources of computing device 900 (e.g., hardware and software resources associated with providing image recognition services) may be allocated to clients upon request.

[0094]

[0101] In one example, computing device 900 may include processing logic 902, a bus interface module 908, a memory 910, and a network interface module 912. These modules may be hardware modules, software modules, or a combination of hardware and software. In some examples, modules may be used interchangeably with component or engine without departing from the scope of this disclosure. Computing device 900 may include additional modules not shown herein. In some implementations, computing device 900 may include fewer modules. In some implementations, one or more modules may be combined into one module. One or more modules may be in communication with each other via a communication channel 914. Communication channel 914 may include one or more buses, meshes, matrices, fabrics, combinations of these communication channels, or some other suitable communication channel.

[0095]

[0102] The processing logic 902 may include one or more integrated circuits, which may include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-chip (SoC), a network processing unit (NPU), a processor configured to execute instructions, or any other circuit configured to perform logical and floating-point operations. Examples of processors that may be included in the processing logic 1102 may include processors developed by ARM®, MIPS®, AMD®, Intel®, Qualcomm®, etc. In some implementations, a processor may include multiple processing cores, and each processing core may be configured to execute instructions independently of the other processing cores. Furthermore, in some implementations, each processor or processing core may implement multiple processing threads that execute instructions on the same processor or processing core while maintaining logical separation between the multiple processing threads. Such processing threads executing on a processor or processing core may be exposed to software as separate logical processors or processing cores. In some implementations, multiple processors, processing cores, or processing threads executing on the same core may share certain resources, such as, for example, a bus, a level 1 (L1) cache, and / or a level 2 (L2) cache. The instructions executed by the processing logic 902 may be stored on a computer-readable storage medium, for example, in the form of a computer program. The computer-readable storage medium may be non-transitory. In some cases, the computer-readable medium may be part of the memory 910. The processing logic 902 may also include hardware circuitry for performing artificial neural network computations, including, for example, the SoC 302.

[0096]

[0103] Access to the processing logic 902 can be granted to a client to provide a personal assistant service requested by the client. For example, the computing device 900 can host a virtual machine capable of executing an image recognition software application. The image recognition software application, when executed, can access the processing logic 902 to, for example, predict objects contained in an image. As another example, access to the processing logic 902 can also be granted as part of a bare metal instance, where an image recognition software application running on a client device (e.g., a remote computer, a smartphone, etc.) can directly access the processing logic 902 to perform recognition of an image.

[0097]

[0104] Memory 910 may include volatile or nonvolatile, or both volatile and nonvolatile types of memory. Memory 910 may include, for example, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and / or some other suitable storage medium. In some cases, some or all of memory 910 may be internal to computing device 900, while in other cases, some or all of memory may be external to computing device 900. Memory 910 may store an operating system including executable instructions that, when executed by processing logic 902, provide an execution environment for executing instructions that provide network functionality for computing device 900. Memory 910 may also store software applications, for example, for performing artificial neural network calculations. For example, memory 910 may store software routines associated with calculating Equations 1-4 above. If processing logic 902 is in the form of an FPGA, memory 910 may store netlist data representing various logic circuit components of processing logic 902.

[0098]

[0105] The bus interface module 908 can enable communication with external entities, such as a host device and / or other components within the computing system, via an external communication medium. The bus interface module 908 may include a physical interface for connecting to a cable, socket, port, or other connection to the external communication medium. The bus interface module 908 may further include hardware and / or software for managing incoming and outgoing transactions. The bus interface module 908 can implement a local bus protocol, such as a Peripheral Component Interconnect (PCI)-based protocol, Non-Volatile Memory Express (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment (SATA), Parallel ATA (PATA), some other standard bus protocol, or a proprietary bus protocol. The bus interface module 908 may include the physical layer for any of these bus protocols, including connectors, power management, and error handling, among other things. In some implementations, the computing device 900 may include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules may implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols.

[0099]

[0106] The network interface module 912 may include hardware and / or software for communicating with a network. The network interface module 912 may include, for example, a physical connector or port for a wired connection to the network and / or an antenna for wireless communication to the network. The network interface module 912 may further include hardware and / or software configured to implement a network protocol stack. The network interface module 912 may communicate with the network using network protocols such as TCP / IP, InfiniBand, RoCE, the Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless protocol, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), Token Ring, Frame Relay, High-Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and / or Point-to-Point Protocol (PPP), among others. In some implementations, the computing device 900 may include multiple network interface modules, each configured to communicate with a different network. For example, in these implementations, the computing device 900 may include a network interface module for communicating with a wired Ethernet network, a wireless 802.11 network, a cellular network, an InfiniBand network, etc. In some embodiments, the computing device 900 may receive a set of parameters, such as the aforementioned weight vectors for generating forget gate coefficients, input coefficients, output coefficients, etc., from a server via the network interface module 912.

[0100]

[0107] The various components and modules of computing device 900 described above may be implemented as discrete components, as a system-on-chip (SoC), as an ASIC, as an NPU, as an FPGA, or any combination thereof. In some embodiments, the SoC or other component may be communicatively coupled to another computing system to provide various services such as traffic monitoring, traffic shaping, computing, etc. In some embodiments of the present technology, the SoC or other component may include multiple subsystems as disclosed herein.

[0101]

[0108] The modules described herein may be software modules, hardware modules, or a suitable combination thereof. If the modules are software modules, they may be embodied on a non-transitory computer-readable medium and processed by any processor of the computer systems described herein. Note that the described processes and architectures may be executed in either real-time or asynchronous mode, prior to any user interaction. The modules may be organized in the manner suggested in FIG. 9, and / or the functionality described herein may be provided by one or more modules existing as separate modules, and / or the module functionality described herein may be spread across multiple modules.

[0102]

[0109] The specification and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense. It will, however, be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the claims.

[0103]

[0110] Other variations are within the spirit of the present disclosure. Accordingly, while the disclosed techniques are susceptible to various modifications and alternative constructions, several illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the disclosure to the particular form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents that are within the spirit and scope of the present disclosure, as defined by the appended claims. Some exemplary embodiments are described by the following clauses. 1. A method for processing multiple data sets in a multi-layer neural network, the multi-layer neural network comprising a first neural network layer, a second neural network layer that processes outputs generated by the first neural network layer, and a third neural network layer that processes outputs generated by the second neural network layer, the method comprising: storing the first data set in a memory; processing the first data set from the memory to generate a first layer output for the first data set using a first neural network layer; storing the first layer output of the first dataset in a memory; Deleting the first data set from the memory; adding a second dataset to a first layer output of a first dataset stored in a memory; processing a second data set from the memory using a first neural network layer to generate a first layer output for the second data set; adding the first layer output of the second dataset to the first layer output of the first dataset stored in the memory; Deleting the second data set from the memory; processing the first layer output of the first data set and the first layer output of the second data set from the memory to generate, using a second neural network layer, a second layer output of the first data set and a second layer output of the second data set, respectively; storing the second layer output of the first data set and the second layer output of the second data set in a memory; removing the first layer output of the first data set and the first layer output of the second data set from the memory; adding a third dataset to the second layer output of the first dataset and the second layer output of the second dataset stored in the memory; processing a third dataset from the memory to generate a first layer output for a third dataset after a second layer output for the first dataset and a second layer output for the second dataset have been generated using the first neural network layer; storing the first layer output of the third dataset in a memory; Deleting the third data set from the memory; and adding a fourth dataset to the second layer output of the first dataset, the second layer output of the second dataset, and the first layer output of the third dataset stored in the memory; processing the fourth data set from the memory to generate a first layer output for the fourth data set using the first neural network layer; adding the first layer output of the fourth dataset to the first layer output of the third dataset, the second layer output of the first dataset, and the second layer output of the second dataset stored in the memory; deleting the fourth data set from the memory; and processing the first layer output of the third data set and the first layer output of the fourth data set to generate, using a second neural network layer, a second layer output of the third data set and a second layer output of the fourth data set, respectively; adding the second layer output of the third dataset and the second layer output of the fourth dataset to the second layer output of the first dataset and the second dataset stored in the memory; processing the second layer output of the first dataset, the second layer output of the second dataset, the second layer output of the third dataset, and the second layer output of the fourth dataset from the memory to generate, using a third neural network layer, a third layer output of the first dataset, a third layer output of the second dataset, a third layer output of the third dataset, and a third layer output of the fourth dataset, respectively; storing the third layer output of the first data set, the third layer output of the second data set, the third layer output of the third data set, and the third layer output of the fourth data set in a memory; A method comprising: 2. The method of clause 1, wherein the second layer output of the first dataset and the second layer output of the second dataset are stored in memory by at least overwriting the first layer output of the first dataset stored in memory. 3. The method of any preceding clause, further including controlling the memory to provide space for simultaneously storing a fourth dataset, a first layer output of the fourth dataset, a first layer output of the third dataset, a second layer output of the first dataset, and a second layer output of the second dataset. 4. The method of any preceding paragraph, wherein the data size of the first layer output of the first dataset is greater than the data size of the second layer output of the first dataset, and the data size of the second layer output of the first dataset is greater than the data size of the third layer output of the first dataset. 5. A method for processing data, comprising: configuring a computational engine in a first configuration to implement a first layer of a neural network; processing first data associated with the first context using a computational engine in the first configuration to generate a first layer output for the first context; configuring the computational engine in the second configuration to implement a second layer of the neural network; processing the first data of the first context using a computational engine in the second configuration to generate a second layer output of the first context; switching the calculation engine from the second configuration back to the first configuration; processing second data associated with the second context using a computational engine in the first configuration to generate a first layer output for the second context; configuring the computation engine to return to a second configuration to implement a second layer of the neural network; processing the first layer output of the second context in the second configuration to generate a second layer output of the second context; configuring the computational engine in the third configuration to implement a third layer of the neural network; processing the second layer output of the first context and the second layer output of the second context using a computation engine in the third configuration to generate a third layer output of the first context and a third layer output of the second context; A method comprising: 6. processing third data related to the third context using a computational engine in the first configuration to generate a first layer output for the third context; processing the first layer output of the third context and the first layer output of the first context using a computation engine in the second configuration to generate a second layer output of the third context and a second layer output of the first context, respectively; 6. The method of clause 5, further comprising: 7. storing the first layer output of the first context and the first layer output of the third context in a memory device; configuring a first portion of the computational engine to receive as input a first layer output of a first context from a memory device; configuring a second portion of the computational engine to receive as input the first layer output of the third context from the memory device; processing a first layer output of a first context using a first portion of a computational engine; using a second portion of the computational engine to process the first layer output in a third context; and 6. The method of claim 5, further comprising: 8. The method of clause 7, wherein the processing of the first layer output of the first context using the first portion of the computational engine and the processing of the first layer output of the third context using the second portion of the computational engine are performed in parallel. 9. processing fourth data related to the fourth context using a computational engine in the first configuration to generate a first layer output for the fourth context; processing the first layer output of the second context and the first layer output of the fourth context using a computation engine in the second configuration to generate a second layer output of the second context and a second layer output of the fourth context, respectively; 9. The method of any of clauses 5 to 8, further comprising: 10. Storing the first layer output of the second context and the first layer output of the fourth context in a memory device; configuring a first portion of the computational engine to receive as input the first layer output of the second context from the memory device; configuring a second portion of the computational engine to receive as input the first layer output of the fourth context from the memory device; processing the first layer output of the second context using a first portion of the computational engine; using a second portion of the computational engine to process the first layer output of the fourth context; and 9. The method of clause 9, further comprising: 11. The method of clause 10, wherein the processing of the first layer output of the second context using the first portion of the computational engine and the processing of the first layer output of the fourth context using the second portion of the computational engine are performed substantially in parallel. 12. Storing the second layer output of the first context in a memory device; storing second data associated with the second context in addition to the second layer output of the first context in the memory device; storing the first layer output of the second context in the memory device along with the second layer output of the first context and second data related to the second context; 12. The method of any of clauses 5 to 11, further comprising: 13. Overwriting at least a portion of the second data associated with the second context or the first layer output of the second context stored in the memory device with the second layer output of the second context. 13. The method of clause 12, further comprising: 14. Configuring a third portion of the computational engine to receive as input the second layer output of the first context from the memory device; configuring a fourth portion of the computational engine to receive as input the second layer output of the second context from the memory device; processing the second layer output of the first context using a third portion of the computational engine; and using a fourth portion of the computational engine to process the second layer output in the second context; and 14. The method of clause 13, further comprising: 15. The method of clause 14, wherein the processing of the second layer output of the first context using the third portion of the computational engine and the processing of the second layer output of the second context using the fourth portion of the computational engine are performed substantially in parallel. 16. The method of any of clauses 5 to 15, wherein processing first data associated with the first context using a computational engine in the first configuration to generate a first layer output for the first context includes performing one or more convolution computations between the first data and a set of weights associated with the first layer of the neural network. 17. Processing the results of the one or more convolutional computations by the activation function engine to generate a first layer output for the first context. The method of clause 16 further includes 18. A memory device configured as a state buffer; a computational engine coupled to a memory device and comprising an array of processing elements, each of the array of processing elements comprising an adder and a multiplier; An integrated circuit comprising: The calculation engine is Retrieving a first set of weights associated with a second layer of the neural network from a state buffer; performing a first set of additions and multiplications between the first set of weights and the first data to generate a second layer output of the first data; retrieving a second set of weights associated with the first layer of the neural network from the state buffer after the second layer output of the first data is generated; performing a second set of additions and multiplications between the second set of weights and the second data to generate a first layer output of the second data; performing a third set of additions and multiplications between the first set of weights and the first layer output of the second data to generate a second layer output of the second data; Retrieving a third set of weights associated with a third layer of the neural network from the state buffer; and performing a fourth set of additions and multiplications between the third set of weights and the second layer output of the first data to generate a first result from processing the first data; and performing a fifth set of additions and multiplications between the third set of weights and the second layer output of the second data to generate a second result of processing the second data; and configured to: a fourth set of additions and multiplications and a fifth set of additions and multiplications are performed by different portions of the array of processing elements; Integrated circuit. 19. The calculation engine storing the second layer output of the first data in a state buffer; performing a fourth set of additions and multiplications between the third set of weights and the second layer output of the first data obtained from the state buffer; 19. The integrated circuit of clause 18, further configured to: 20. The integrated circuit of clause 18 or 19, wherein the computational engine is configured to perform the fourth set of additions and multiplications and the fifth set of additions and multiplications substantially in parallel. 21. Determining a schedule for performing neural network computations for a first context and a second context, wherein performing the neural network computations for each context includes performing a sequence of computational tasks for the respective context, each sequence of computational tasks including a first computational task using a first neural network layer, a second computational task using a second neural network layer, and a third computational task using a third neural network layer, wherein the third computational task of one context is configured to process an output of the second computational task of one context, and the second computational task of one context is configured to process an output of the first computational task of one context; Determining the schedule determining a first execution batch including a third computation task of a first context and a third computation task of a second context, the first execution batch being determined based on a state buffer having sufficient capacity to hold input data and output data of the third computation task of the first context and the third computation task of the second context; determining, determining whether the state buffer has sufficient capacity to hold input data and output data of the first context and the second computational task of the second context; In response to determining that the state buffer does not have sufficient capacity to hold input data and output data of the first context and the second computational task of the second context, determining a second execution batch to be executed before the first execution batch and a third execution batch to be executed before the second execution batch, wherein the second execution batch includes a second computational task of a first context and the third execution batch includes a second computational task of a second context, and the second execution batch and the third execution batch are determined based on a state buffer having sufficient capacity to hold output data of the second computational task of the second context and input data and output data of the second computational task of the first context; determining a fourth execution batch to be executed before the second execution batch, the fourth execution batch including a first computational task of a first context, the fourth execution batch being determined based on a state buffer having sufficient capacity to hold input data and output data of the first computational task of the first context and output data of the second computational task of the second context; executing a third execution batch; executing a fourth execution batch after the third execution batch; executing the second execution batch after the fourth execution batch; executing the first execution batch after the second execution batch; A method comprising: 22. Determining whether the state buffer has sufficient capacity to hold a set of weights for the first context and the second computational task of the second context, in addition to input data and output data for the first context and the second computational task of the second context. further comprising determining a second execution batch to be executed before the first execution batch and a third execution batch to be executed before the second execution batch is based on the state buffer having insufficient capacity to hold the sets of weights, input data, and output data of the second computational tasks of the first context and the second context; Section 21 Method. 23. Maximizing a first number of second computation tasks to be included in a second execution batch and a second number of second computation tasks to be included in a third execution batch, so that the total data size of the input data and output data of the first number of second computation tasks and the output data of the second number of second computation tasks is less than the capacity of the state buffer when the second execution batch is executed. 23. The method of clause 21 or 22, further comprising: 24. Determining a fifth execution batch to be executed before the third execution batch, where the third execution batch includes a first computational task of a second context, and the fifth execution batch is determined based on the state buffer having sufficient capacity to hold input data and output data of the first computational task of the second context; executing the fifth execution batch before the third execution batch; 24. The method of any of clauses 21 to 23, further comprising: 25. Determining a set of computational tasks to be executed, the set of computational tasks including a first computational task, a second computational task, a third computational task, and a fourth computational task, the third computational task and the fourth computational task generating input data for the first computational task and input data for the second computational task, respectively; determining a first execution batch including the first computational task and the second computational task based on the memory device having sufficient capacity to hold input and output data of both the first computational task and the second computational task; determining a second execution batch including at least a third computational task to be executed before the first execution batch; determining whether to include the fourth computational task in the second execution batch based on whether the memory device has sufficient capacity to hold input data and output data for both the third computational task and the fourth computational task; executing a second execution batch; executing the first execution batch after the second execution batch; A method comprising: 26. Determining the second batch to be executed determining that the memory device does not have sufficient capacity to hold the input data and output data of both the third computational task and the fourth computational task, but has sufficient capacity to hold the input data and output data of the third computational task and the output data of the fourth computational task; determining a second execution batch including a third computational task; determining a third execution batch including a fourth computational task to be executed before the second execution batch; wherein the method comprises: Executing the third execution batch before the second execution batch The method of clause 25, including 27. The set of computational tasks further includes a fifth computational task and a sixth computational task that generate input data for the third computational task and the fourth computational task, respectively; The method is determining a fourth execution batch including a fifth computational task; determining a fifth execution batch including a sixth computational task; executing a fourth execution batch before the second execution batch; executing the fifth execution batch before the third execution batch; 27. The method of clause 26, further comprising: 28. Determining a fourth execution batch to be executed between the third execution batch and the second execution batch based on determining that the memory device does not have sufficient capacity to store the output data of the fifth calculation task, the output data of the sixth calculation task, and the input data and output data of the fourth calculation task, and that the memory device has sufficient capacity to store the output data of the fourth calculation task and the input data and output data of the fifth calculation task. The method of clause 27, further comprising: 29. Determining a fourth execution batch to be executed before the third execution batch and before the second execution batch based on determining that the memory device has sufficient capacity to store the output data of the fifth computational task, the output data of the sixth computational task, and the input data and output data of the fourth computational task. 29. The method of clause 27 or 28, further comprising: 30. Determining the second batch to be executed determining a second execution batch including the third computational task and the fourth computational task based on determining that the memory device has sufficient capacity to hold input data and output data for both the third computational task and the fourth computational task; any method set forth in clauses 25 to 29, including 31. The set of computational tasks further includes a fifth computational task and a sixth computational task that generate input data for the third computational task and the fourth computational task, respectively; The method is determining the third computation batch based on whether the memory device has sufficient capacity to hold the input data and output data of both the fifth computation task and the sixth computation task; executing a third execution batch before the second execution batch; The method of clause 30, further comprising: 32. Determining the third batch of execution determining that the memory device does not have sufficient capacity to store the input data and output data of both the fifth computational task and the sixth computational task, but has sufficient capacity to store the input data and output data of the fifth computational task and the output data of the sixth computational task; determining a third execution batch including a fifth computational task; determining a fourth execution batch including a sixth computational task to be executed before the third execution batch; wherein the method comprises: Executing the fourth batch The method of clause 31, including 33. Determining the third batch of execution determining that the memory device has sufficient capacity to store input data and output data for both the fifth computational task and the sixth computational task; determining a third execution batch including a fifth computational task and a sixth computational task; the method of clause 31 or 32, including 34. The method of any of clauses 25 to 33, wherein executing the second execution batch includes executing the first computational task and the second computational task substantially in parallel. 35. The method of any of clauses 25 to 34, wherein the first computational task and the third computational task are for processing first data associated with a first context, and the second computational task and the fourth computational task are for processing second data associated with a second context. 36. The method of any of clauses 25 to 35, wherein the third computational task and the fourth computational task are associated with computation for a first neural network layer, and the first computational task and the second computational task are associated with computation for a second neural network layer that processes data from the first neural network layer. 37. A memory device configured as a state buffer; a computational engine coupled to a memory device; determining a set of computational tasks to be executed on a computational engine, the set of computational tasks including a first computational task, a second computational task, a third computational task, and a fourth computational task, the third computational task and the fourth computational task generating input data for the first computational task and the second computational task, respectively; determining a first execution batch including the first computational task and the second computational task based on the memory device having sufficient capacity to hold input and output data of both the first computational task and the second computational task; determining a second execution batch to be executed before the first execution batch based on whether the memory device has sufficient capacity to hold input data and output data of both the third computational task and the fourth computational task; executing a second execution batch in the calculation engine; executing the first execution batch in the calculation engine after the second execution batch has completed execution; a controller configured to 1. An integrated circuit comprising: 38. The controller determining that the memory device does not have sufficient capacity to hold the input data and output data of both the third computational task and the fourth computational task, but has sufficient capacity to hold the input data and output data of the third computational task and the output data of the fourth computational task; determining that the second execution batch includes a third computational task; determining a third execution batch including a fourth computational task to be executed before the second execution batch; allocating a first memory space in the memory device for storing input data of a fourth computational task and a second memory space in the memory device for storing output data of the fourth computational task; executing a third execution batch on the computation engine by controlling the computation engine to read input data of the fourth computation task from the first memory space and store output data of the fourth computation task in the second memory space; storing output data of the fourth computational task in a memory device during execution of the second execution batch in the computational engine; 38. The integrated circuit of clause 37, further configured to: 39. The set of computational tasks further includes a fifth computational task and a sixth computational task that generate input data for the third computational task and the fourth computational task, respectively; The controller determining a fourth execution batch including a fifth computational task; determining a fifth execution batch including a sixth computational task; executing a fourth execution batch in the calculation engine before executing the second execution batch; executing a fifth execution batch in the calculation engine before executing the third execution batch; 39. The integrated circuit of clause 38, further configured to: 40. The first computational task, the third computational task, and the fifth computational task are associated with a third neural network layer computation, a second neural network layer computation, and a first neural network layer computation of the neural network model for the first context, respectively; the second computational task, the fourth computational task, and the sixth computational task are associated with a third neural network layer computation, a second neural network layer computation, and a first neural network layer computation of the neural network model for the second context, respectively; Section 39 Integrated Circuits.

[0104]

[0111] In the context of describing the disclosed embodiments (particularly in the context of the claims below), the use of the terms “a,” “an,” and “the” and similar referents should be interpreted to cover both the singular and the plural unless otherwise indicated herein or clearly contradicted by context. The terms “comprise,” “have,” “include,” and “including” should be interpreted as open-ended (i.e., meaning “including, but not limited to”) unless otherwise noted. The term “connected” should be interpreted as partially or wholly contained within, connected to, or joined together, even if there is something intervening. The recitation of ranges of values ​​herein is intended only to serve as a shorthand method of individually referring to each individual value falling within the range, unless otherwise indicated herein, and each individual value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The use of any and all examples or exemplary language (e.g., "etc.") provided herein is intended only to better illuminate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0105]

[0112] Selection language such as the phrase "at least one of X, Y, or Z," unless otherwise specified, is to be understood within the context as being used generally to indicate that an item, term, etc. is X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such selection language generally does not and should not imply that a particular embodiment requires that at least one X, at least one Y, or at least one Z, respectively, be present.

[0106]

[0113] Various embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the disclosure. Variations of these embodiments may become apparent to those skilled in the art upon reading the foregoing description. The inventors expect those skilled in the art to utilize such variations as they see fit, and the inventors intend for the present disclosure to be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure unless otherwise indicated herein or clearly contradicted by context.

Claims

1. A method performed by a processor, the method comprising: determining a schedule for performing neural network computations for a first context and a second context, wherein performing the neural network computations for each context includes performing a sequence of computation tasks for the respective context, each sequence of computation tasks including a first computation task using a first neural network layer, a second computation task using a second neural network layer, and a third computation task using a third neural network layer, the third computation task of a context being configured to process an output of the second computation task of the one context, and the second computation task of the one context being configured to process an output of the first computation task of the one context; determining the schedule, determining a first execution batch including the third computational task of the first context and the third computational task of the second context, wherein the first execution batch is determined based on a state buffer having sufficient capacity to hold input data and output data of the first context and the third computational task of the second context; determining, determining whether the state buffer has sufficient capacity to hold input data and output data of the first context and the second computational task of the second context; in response to determining that the state buffer does not have sufficient capacity to hold the input data and the output data of a second computational task of the first context and the second context, determining a second execution batch to be executed before the first execution batch and a third execution batch to be executed before the second execution batch, wherein the second execution batch includes the second computational task of the first context and the third execution batch includes the second computational task of the second context, and the second execution batch and the third execution batch are determined based on the state buffer having sufficient capacity to hold output data of the second computational task of the second context and the input data and the output data of the second computational task of the first context; determining a fourth execution batch to be executed before the second execution batch, the fourth execution batch including the first computational task of the first context, the fourth execution batch being determined based on the state buffer having sufficient capacity to hold input data and output data of the first computational task of the first context and the output data of the second computational task of the second context; determining a fifth execution batch to be executed before the third execution batch, the fifth execution batch including the first computational task of the second context; and executing the third execution batch after the fifth execution batch; executing the fourth execution batch after the third execution batch; executing the second execution batch after the fourth execution batch; executing the first execution batch after the second execution batch; A method comprising:

2. determining whether the state buffer has sufficient capacity to hold a set of weights for the second computational task of the first context and the second context in addition to the input data and the output data of the first context and the second computational task of the second context; further comprising 2. The method of claim 1 , wherein determining a second execution batch to be executed before the first execution batch and a third execution batch to be executed before the second execution batch is based on the state buffer having insufficient capacity to hold the set of weights, the input data, and the output data of the second computational task of the first context and the second context.

3. maximizing a first number of second computational tasks to be included in the second execution batch and a second number of second computational tasks to be included in the third execution batch, such that a total data size of the input data and the output data of the first number of second computational tasks and the output data of the second number of second computational tasks is less than a capacity of the state buffer when the second execution batch is executed. The method of claim 1 further comprising:

4. determining the fifth execution batch to be executed before the third execution batch, the fifth execution batch being determined based on the state buffer having sufficient capacity to hold the input data and the output data of the first computational task of the second context; executing the fifth execution batch before the third execution batch; The method of claim 1 further comprising:

5. A method performed by a processor, the method comprising: determining a set of computational tasks to be performed associated with a neural network layer computation, the set of computational tasks including a first computational task, a second computational task, a third computational task, and a fourth computational task, the third computational task and the fourth computational task generating input data for the first computational task and the second computational task, respectively; determining a first execution batch including the first computational task and the second computational task based on a memory device having sufficient capacity to hold input data and output data of both the first computational task and the second computational task; determining a second execution batch including at least a third computational task to be executed before the first execution batch; determining whether to include the fourth computing task in the second execution batch based on (1) whether the memory device has sufficient capacity to hold input data and output data of both the third computing task and the fourth computing task, or (2) whether the memory device has only sufficient capacity to hold the output data of the third computing task and the input data and the output data of the fourth computing task; determining a third execution batch including the fourth computational task to be executed after the second execution batch in response to determining that the memory device does not have sufficient capacity to hold the input data and the output data of both the third computational task and the fourth computational task, but has sufficient capacity to hold the input data and the output data of the fourth computational task and the output data of the third computational task; executing the second execution batch; executing the third execution batch after the second execution batch; executing the first execution batch after the third execution batch; A method comprising:

6. 6. The method of claim 5, wherein the set of computational tasks further includes a fifth computational task and a sixth computational task that generate the input data for the third computational task and the input data for the fourth computational task, respectively.

7. The method comprises: determining a fourth execution batch including the fifth computational task; determining a fifth execution batch including the sixth computational task; executing the fourth execution batch before the second execution batch; executing the fifth execution batch before the third execution batch; The method of claim 6 further comprising:

8. determining the fourth execution batch to be executed before the third execution batch and before the second execution batch based on determining that the memory device has sufficient capacity to store the output data of the fifth computational task, the output data of the sixth computational task, and the input data and the output data of the fourth computational task. The method of claim 7 further comprising:

9. The method of claim 5 , wherein at least a portion of the first computational task and at least a portion of the second computational task are performed in parallel.

10. 6. The method of claim 5, wherein the first computational task and the third computational task are for processing first data associated with a first context, and the second computational task and the fourth computational task are for processing second data associated with a second context.

11. The method of claim 10 , wherein the first data associated with the first context and the second data associated with the second context are from different data sources.

12. 6. The method of claim 5, wherein the third computational task and the fourth computational task are associated with computation for a first neural network layer, and the first computational task and the second computational task are associated with computation for a second neural network layer that processes data from the first neural network layer.

13. determining a first scheduled execution time for the first execution batch; determining a second scheduled execution time for the second execution batch after determining the first scheduled execution time; the second execution batch is executed at the second scheduled execution time; The method of claim 5 , wherein the first execution batch is executed at the first schedule execution time after the second schedule execution time.

14. The method of claim 5 , wherein the set of computational tasks is performed by a computational engine that is part of the same chip as the memory device.

15. a memory device configured as a state buffer; a computational engine coupled to the memory device; A control circuit comprising: determining, in the computation engine, a set of computational tasks to be performed associated with neural network layer computations, the set of computational tasks including a first computational task, a second computational task, a third computational task, and a fourth computational task, the third computational task and the fourth computational task generating input data for the first computational task and the second computational task, respectively; determining a first execution batch including the first computational task and the second computational task based on the memory device having sufficient capacity to hold input data and output data of both the first computational task and the second computational task; determining a second execution batch to be executed before the first execution batch based on whether (1) the memory device has sufficient capacity to hold input data and output data of both the third computational task and the fourth computational task, or (2) whether the memory device has only sufficient capacity to hold the output data of the third computational task and the input data and the output data of the fourth computational task; determining a third execution batch including the fourth computational task to be executed after the second execution batch in response to determining that the memory device does not have sufficient capacity to hold the input data and the output data of both the third computational task and the fourth computational task, but has sufficient capacity to hold the input data and the output data of the fourth computational task and the output data of the third computational task; executing the second execution batch on the computation engine; executing the third execution batch on the computation engine after the second execution batch has completed execution; executing the first execution batch in the computing engine after the second execution batch has completed execution; a control circuit configured to:

1. An integrated circuit comprising:

16. The control circuit allocating a first memory space in the memory device for storing the input data of the third computing task and a second memory space in the memory device for storing the output data of the third computing task, wherein the second execution batch is executed on the computing engine by controlling the computing engine to read the input data of the third computing task from the first memory space and store the output data of the third computing task in the second memory space; storing the output data of the third computational task in the memory device during execution of the third execution batch in the computation engine; The integrated circuit of claim 15 further configured to:

17. the set of computational tasks further includes a fifth computational task and a sixth computational task for generating the input data for the third computational task and the input data for the fourth computational task, respectively; The control circuit determining a fourth execution batch including the fifth computational task; determining a fifth execution batch including the sixth computational task; executing the fourth execution batch on the computation engine before executing the second execution batch; executing the fifth execution batch in the calculation engine before executing the third execution batch; The integrated circuit of claim 16 further configured to:

18. the first computational task, the third computational task, and the fifth computational task are associated with a third neural network layer computation, a second neural network layer computation, and a first neural network layer computation of a neural network model for a first context, respectively; the second computational task, the fourth computational task, and the sixth computational task are associated with a third neural network layer computation, a second neural network layer computation, and a first neural network layer computation of the neural network model for a second context, respectively; 18. The integrated circuit of claim 17.

Images