Parameter cache for neural network accelerator

Abstract

Methods and systems including computer programs encoded on computer storage media. In one aspect, a method includes obtaining data specifying one or more neural networks to be deployed on a neural network hardware accelerator, each of the one or more neural networks having a respective set of parameters, and the neural network hardware accelerator having one or more memories having a memory capacity; determining a maximum amount of the memory capacity to be used at any one time during processing of any one of the one or more neural networks by the neural network hardware accelerator; identifying a subset of parameters of the one or more neural networks that consumes an amount of memory that is less than a difference between the memory capacity and the determined maximum amount of the memory capacity; and storing the identified subset of parameters.

Description

Technical Field

[0001] This specification relates to parameter caching techniques for neural network layer computation on neural network hardware accelerators. Background Technology

[0002] A neural network is a machine learning model that uses one or more layers to generate an output (e.g., classification) from received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as input to another layer in the network—e.g., the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters.

[0003] The corresponding parameter set is stored in the memory of the neural network hardware accelerator. When performing parameter transformation to the computational unit of the neural network hardware accelerator, such as neural network layer computation for convolutional layer computation, it is performed by multiplying the input tensor on the computational unit of the neural network hardware accelerator with the stored parameter set. Summary of the Invention

[0004] This specification describes a technique for parameter caching in neural network computations within a neural network hardware accelerator. The neural network processing system caches neural network parameters in the neural network accelerator before executing neural network layers. Specifically, the neural network processing system pre-caches the neural network parameters into the available wide memory of the computational units within the neural network accelerator to utilize the abundant memory of the neural network accelerator, thereby improving the efficiency of the neural network system.

[0005] More specifically, the technique involves determining the available memory capacity of the computing units in a neural network accelerator. The system identifies a subset of neural network parameters that are smaller than the available memory capacity of the computing units. The identified subset of neural network parameters is then transferred from data memory to the computing units and stored in the wide memory of the computing units in the hardware accelerator for use in neural network layer computation. In this technique, parameter pre-caching is differentiated based on the properties of the neural network layers. For fully connected neural network layers, the technique splits the parameters based on the amount of data in the corresponding computing unit and the available memory capacity of the corresponding computing unit. For convolutional neural network layers, the technique differentiates the parameters based on their correspondence with the convolutional layer and transfers the differentiated parameters one by one from data memory to the wide memory of the computing units in the hardware accelerator.

[0006] An innovative aspect of the subject matter described in this specification can be embodied in a method comprising: obtaining data for one or more neural networks to be deployed on a neural network hardware accelerator, each of the one or more neural networks having a corresponding set of parameters, and the neural network hardware accelerator having one or more memories having memory capacity; determining a maximum amount of the memory capacity to be used at any time during processing of any of the one or more neural networks by the neural network hardware accelerator; identifying a subset of parameters of the one or more neural networks, the amount of memory consumed by the subset of parameters being less than the difference between the memory capacity and the determined maximum amount of the memory capacity; and storing the identified subset of parameters in at least one of the one or more memories of the neural network hardware accelerator.

[0007] Another innovative aspect of the subject matter described in this specification can be embodied in a neural network hardware accelerator comprising a plurality of hardware computing units, each of which is a processing element having one or more memories and configured to perform neural network computations. The hardware computing units are further configured to perform operations including: executing a plurality of instructions from the neural network hardware accelerator by a processor of the hardware computing unit; accessing one or more memories of the hardware computing unit to retrieve an identified subset of parameters; sending the identified subset of parameters to one or more memories of other hardware computing units of the neural network hardware accelerator via a ring bus; and storing identified subsets of parameters from other hardware computing units of the neural network hardware accelerator to one or more memories of the hardware computing unit via a ring bus.

[0008] Another innovative aspect of the subject matter described in this specification may be one or more neural networks including fully connected layers, wherein the parameters of the fully connected layers are in an identified subset, wherein the accelerator is configured to distribute the processing of the fully connected layers across a first subset of hardware computing units, and wherein storing the identified subset comprises storing a corresponding partition of the parameters of the fully connected layers in one or more memories of each hardware computing unit in the first subset.

[0009] Another innovative aspect of the subject matter described in this specification may be one or more neural networks including convolutional layers, wherein the parameters of the convolutional layers are in an identified subset, wherein the accelerator is configured to distribute the processing of the convolutional layers across a second subset of hardware computing units, and wherein storing the identified subset comprises storing corresponding copies of the parameters of the convolutional layers in one or more memories of each hardware computing unit in the second subset.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a block diagram of an exemplary neural network processing system.

[0012] Figure 2 This is a diagram of an exemplary neural network hardware accelerator used for neural network computation.

[0013] Figure 3 This is an exemplary flowchart of the process for performing parameter caching on a neural network hardware accelerator.

[0014] Figure 4 This is an exemplary flowchart of the process for performing parameter caching for computation of fully connected neural network layers on a neural network hardware accelerator.

[0015] Figure 5 This is an exemplary flowchart of a process for performing parameter caching for convolutional neural network layer computation on a neural network hardware accelerator.

[0016] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0017] Neural network hardware accelerators are often used to more efficiently perform operations or computations for neural network layers.

[0018] High-performance hardware accelerators, such as Tensor Processing Units (TPUs), have one or more wide memories located on the accelerator's compute units. Each wide memory may have local storage capacity, such as 8MB, to cache instructions and parameter data used for neural network layer computations. Typically, a small portion of the capacity of the wide memory on the compute unit is first reserved for the inference executable program of the model, and then the parameter data is stored in any remaining space. For example, before neural network computations are executed, the neural network model parameters need to be stored in the wide memory of the compute unit. In many cases, the model parameters are loaded from the host interface of the neural network processing system into the local memory of the corresponding compute unit—e.g., the compute unit itself. When executing a neural network layer that requires the corresponding model parameters, the neural network processing system issues instructions to transfer the model parameters from the parameter memory to the receiver compute unit of the hardware accelerator via a ring bus. The receiver compute unit then copies the model parameters from the ring bus to its wide memory based on the issued operation instructions.

[0019] The aforementioned parameter transformations may not provide optimal efficiency for neural network processing systems and optimal memory utilization for neural network hardware accelerators. There may be abundant, wide memory capacities of the accelerator's computational units that are not fully utilized during neural network layer computations. Furthermore, external parameter memory may have lower bandwidth compared to the wide memory on the computational units, potentially causing computation to stall while waiting for data transformations.

[0020] This specification describes a parameter caching technique for high-performance neural network layer computation on a hardware accelerator. This parameter caching technique includes obtaining data for one or more neural network layers specified to be deployed on one or more computing units of the neural network accelerator, determining the maximum amount of wide memory capacity that will be used at any time during the processing of any of the one or more neural network layers, and identifying a subset of parameters that consume less memory compared to the available wide memory capacity on the computing unit. Model parameters are stored in the wide memory of the hardware computing unit of the neural network hardware accelerator before neural network computation is executed. More specifically, the wide memory located on the computing unit of the accelerator can be a temporary storage memory allocated by the compiler. The accelerator compiler can be configured to add a small amount of executable file within the model, which writes a specific amount of parameter data to the wide memory of the computing unit on the accelerator before running inference. This parameter caching technique enables faster inference speeds compared to fetching parameter data from external parameter memory as previously described.

[0021] Figure 1 A block diagram of an exemplary neural network processing system 100 is shown. Figure 1 As shown, the neural network processing system 100 includes a neural network accelerator 110, a host interface 120, a main memory 130, a neural network implementation engine 140, and an output 150. The neural network processing system 100 may include two or more memories to store data and instructions used for neural network computation.

[0022] In the neural network processing system 100, the host interface 120 receives parameters and instructions and delivers them to the neural network hardware accelerator 110 and the main memory 130. Parameters may include weights and biases of neural network layers. The host interface 120 may also send instructions to the neural network hardware accelerator 110, which executes instructions to control the neural network hardware accelerator 110 to perform neural network computations. In some embodiments, the transmitted instructions regulate the data flow in the neural network hardware accelerator, for example, how the feature input set and filter input set flow through the circuitry.

[0023] Neural network processing system 100 is a system that performs neural network computations using neural network hardware accelerator 110. Neural network hardware accelerator 110 is an integrated circuit for performing neural network computations and includes one or more computational units that perform matrix multiplication within the hardware accelerator. The computational units of neural network hardware accelerator 110—e.g., hardware processing units—may also be configured to perform vector multiplication. In some embodiments, neural network accelerator 110 receives task instructions and performs a multiplication operation that may include multiplying activations by weights or other parameters to produce an output, such as partially summing or outputting activations to produce an output 150. In some other embodiments, neural network hardware accelerator 110 processes at least one of tensor operation instructions or direct memory access operation instructions. In this example, neural network hardware accelerator 110 may be configured to execute at least one tensor operation instruction and at least one direct memory access operation instruction to perform tensor computations including multiple linear algebraic computations. In the following, references are made to… Figure 2 The exemplary neural network hardware accelerator 110 and hardware computing unit are described in more detail.

[0024] In the neural network processing system 100, the neural network accelerator 110 is configured to process neural network computations, such as matrix multiplication or vector computations, by executing instructions or parameters transferred from the host interface 120 or stored in the main memory 130. The main memory 130 may include one or more memory banks or cells for storing parameters, activations, and weights used to process neural network computations for a given neural network layer. In some embodiments, the main memory 130 is one or more volatile memory cells. In some other embodiments, the main memory 130 is one or more non-volatile cell memory cells, such as, for example, read-only memory (ROM) and / or electrically erasable programmable read-only memory (EEPROM). The main memory 130 may also be another form of computer-readable medium, such as a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state memory device, or a device array including devices in a storage area network or other configuration. In the main memory 130, parameters or activations can be preloaded into corresponding memory address locations corresponding to elements of an input tensor to perform computations for a neural network layer. Similarly, weights and other parameters can be preloaded into main memory 130 at the corresponding memory address locations of the weight tensor or other elements.

[0025] Typically, when the neural network accelerator 110 executes one or more instructions for accessing a specific element of a tensor, it determines the memory address of the element, allowing the neural network hardware accelerator 110 to access main memory 130 to read data representing the value of the specific element. For example, parameters stored in main memory 130 may include parameters for computation of fully connected neural network layers or parameters for computation of convolutional neural network layers. To implement a neural network on the neural network hardware accelerator 110, the neural network processing system 100 includes a neural network implementation engine 140, which is implemented as one or more computer programs on one or more computers in one or more physical locations. The neural network implementation engine 140 generates instructions that, when executed by the neural network hardware accelerator 110, cause the neural network hardware accelerator 110 to perform operations specified by the neural network to generate a neural network output 150.

[0026] Figure 2 A block diagram of an exemplary neural network hardware accelerator 110 for neural network computation is shown. The neural network hardware accelerator 110 typically includes distributed hardware computing units for neural network computation. Each distributed computing unit in the set of computing units 208 has one or more wide memories 210. The neural network hardware accelerator also includes an instruction bus 212, a mesh bus 214, and a ring bus 216 for translating instructions and parameters to the set of hardware computing units 208, respectively. Figure 2 As shown, main memory 130 includes data memory 204 and instruction memory 206 to store parameters and instructions transferred from host interface 120. Main memory 130 is configured to execute one or more instructions related to neural network layer computations in neural network hardware accelerator 110, including instructions stored in instruction memory 206. Instruction memory 206 may store one or more machine-readable instructions executable by one or more processors of main memory 130. Data memory 204 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations occurring within neural network hardware accelerator 110.

[0027] As previously Figure 1As described above, host interface 120 is connected to main memory 130 and configured to provide computation unit instructions and parameters to main memory 130. In some embodiments, computation instructions can be provided to one or more hardware computation units in neural network computing hardware accelerator 110 via instruction bus 212, and model parameters can be provided to one or more hardware computation units in neural network hardware accelerator 110 via ring bus 216. In some embodiments, computation instructions and model parameters are initially received by main memory 130 from host interface 120 and stored in instruction memory 206 and data memory 204, respectively, for later execution by neural network hardware accelerator 110.

[0028] Typically, the hardware computing units in computing unit set 208 are the distributed computing core within neural network hardware accelerator 110 and the focus for neural network computations—e.g., tensor computations. Each computing unit is an independent piece of computing hardware that collaborates with other computing units in computing unit set 208 to accelerate neural network computations across one or more neural network layers. Although computing units can share the execution of tensor computations associated with a given computing unit instruction, a single computing unit is a self-contained computing component configured to perform a subset of tensor computations independently relative to other corresponding computing units in computing unit set 208. Figure 2 An exemplary N×2 computing unit layout is presented, wherein 2N-4 edge computing units are computing units with only three adjacent computing units, and the four corner computing units are computing units with two adjacent computing units. Regarding the data flow method via mesh bus 214, typically, each input activation arriving at a particular computing unit via mesh bus 214 must be submitted to one or more wide memories 210 of the computing unit.

[0029] Instruction bus 212 originates from main memory 130 and provides communication coupling via a bus data path that connects computing units in computing unit set 208 to main memory 130 in a ring configuration. In some embodiments, main memory 130 broadcasts one or more computing instructions via instruction bus 212.

[0030] Within the compute unit set 208, a bus station associated with the instruction bus 212 may exist, which is configured to check header part diagrams to determine the instruction type. Compute instructions can be received by the compute units and subsequently written to the compute unit's instruction buffer before execution. The instruction buffer may include a first-in, first-out (FIFO) control scheme that prioritizes the consumption of one or more related compute instructions. Therefore, under this FIFO control scheme, compute instructions of the same type / subtype will always be executed in the order in which the compute unit instructions arrive at the instruction bus.

[0031] Mesh bus 214 provides data communication paths between adjacent computing units in computing unit set 208 in both horizontal and vertical dimensions. In various embodiments, mesh bus 214 can be used to transfer input parameters, such as activation values, between one or more wide memories 210 in adjacent computing units. Figure 2 As shown, the mesh bus 214 is not configured to allow input data to be forwarded directly to non-adjacent computing units.

[0032] The ring bus 216 originates from the main memory 130 and provides communication coupling via a bus data path that connects the set of computing units 208 in the ring back to the main memory 130. In various embodiments, the ring bus 216 typically connects or couples the wide memory 210 cells of all computing units in the set of computing units 208. Therefore, the payload width of the ring bus 216 corresponds to the width of the wide memory 210 disposed within each computing unit. Furthermore, the ring bus 216 may include a bitmap header indicating the computing unit that needs to consume computing instructions and model parameters transmitted via the ring bus 128.

[0033] In some implementations, main memory 130 can be used to pop data from a ring bus station and forward the data to the ring bus station in the next compute unit in the ring. In various implementations, the wide memory 210 of each compute unit in the neural network hardware accelerator 110 is a single-port, single-set memory within each compute unit of the compute unit set 208. If an instruction in the bitmap header requires such an action, main memory 130 can cause data to be committed to one or more wide memory 210 units of the compute unit. The addresses of the one or more wide memory 210 units where data needs to be written can be generated by direct memory access operation instructions within the specific compute unit.

[0034] In various implementations, each computing unit in the computing unit set 208 can be either a data producer or a data consumer. When a computing unit is a data producer, it reads data from one or more of its wide memories 210 and multicasts the data via the ring bus 216 for consumption by one or more adjacent computing units. When a computing unit is a data consumer, it receives data and writes the data into one or more of its wide memories 210, and forwards the data for consumption by one or more other computing units. Regarding data movement via the ring bus 216, typically only one producer / master data transition occurs on the ring bus 216 at any given time. The execution order of direct memory access operation instructions in all computing units (e.g., a FIFO control scheme) ensures that only one producer / master data transition exists on the ring bus 216 at a given time to avoid data transition conflicts.

[0035] In various implementations, when there are local multicast groups connected via ring bus 216 that do not have overlapping areas on ring bus 216, it is necessary to ensure that only one data producer / master device exists on ring bus 216 at a given time. A key requirement of this dual-master multicast method is that different multicast groups must not be allowed to see each other's data packets, as packet overlap can occur and lead to one or more data calculation errors.

[0036] In various implementations, the neural network processing system 100 performs the following operations: Host interface 120 provides one or more instructions to main memory 130 and defines one or more direct memory access operations occurring for a given neural network computation. The descriptors associated with the instructions fed to main memory 130 include information necessary to facilitate large-scale neural network computation associated with the hardware computing unit set 208. Typically, main memory 130 receives from host interface 120 computation unit instructions or model parameters (i.e., input activations, computation unit instructions, and model weights) for performing neural network computations on specific layers of the neural network. Main memory 130 can then cause the instructions or model parameters to be multicast to the computing units in a data stream manner defined by the instructions. As described above, the computing unit consuming the instructions can then initiate the broadcast of new / subsequent instructions to another computing unit based on bitmap data in the instruction header.

[0037] In various implementations, model parameters may be stored in one or more wide memories 210 of the computing unit before neural network computation can be performed in the neural network hardware accelerator 110. In some implementations, when the host interface 120 transmits parameters to the neural network hardware accelerator 110, the main memory 130 receives the parameters and writes them to its data memory 204. When performing neural network computation on a specific neural network layer, the host interface 120 will provide issued instructions, such as RinginFeed instructions. The issued instructions initiate operations to transmit parameters from the data memory 204 to a specific computing unit within the computing unit set 208 via the ring bus 216 and the mesh bus 214. The specific receiver computing unit then issues a direct memory access operation instruction to receive the transmitted parameters from the ring bus 216 and write them to one or more wide memories of the receiver computing unit.

[0038] Regarding data storage in the neural network hardware accelerator 110, each computing unit in the computing unit set 208 can store a subset of the input instructions and parameters required to compute a subset of the output activations assigned to a specific computing unit. Neural network computation begins when the required input activations, parameters / weights, and computation instructions are available in the computing unit. Computation ends in that computing unit when all computational operations defined by the instruction set are completed and the pre-activation function is applied to the result of the neural network computation (i.e., the output activation).

[0039] Typically, as previously described, parameters need to be transferred and stored in one or more wide memories 210 of the computing unit before executing neural network computation instructions. Parameters can be loaded into data memory 204 within main memory 130 via host interface 120. When executing a neural network layer that requires parameters, the neural network layer issues instructions to transfer model parameters from data memory 204 to one or more specific computing units via ring bus 216.

[0040] This application proposes a method for caching parameters into a neural network hardware accelerator 110 before executing any input neural network layer. This parameter cache leverages the abundant bandwidth of the wide memory 210 to improve system efficiency and utilization of high-performance neural network computation, and is described in more detail in the following figures.

[0041] Figure 3 This is an exemplary flowchart of the process for implementing parameter caching on a neural network hardware accelerator. It can be used... Figure 1 and Figure 2 The neural network hardware accelerator 110 is used to execute this example.

[0042] The neural network hardware accelerator 110 obtains data specifying one or more neural network layers to be deployed on one or more computing units of the neural network hardware accelerator 110 (step 302). Each of the one or more neural network layers has a corresponding set of parameters, and the neural network hardware accelerator 110 has one or more memories with memory capacity. The set of parameters may be the weights and biases of the neural network. The memory of the neural network hardware accelerator may be one or more wide memories 210 on the computing units of the computing unit set 208. In some embodiments, the host interface 120 receives instructions to execute specific neural network layers and model parameters on the neural network hardware accelerator 110. The computation instructions and model parameters are transferred to the main memory 130 and stored in the instruction memory 206 and the data memory 204, respectively.

[0043] In the next step, the neural network processing system 100 determines the maximum amount of wide memory 210 capacity that will be used at any time during any of the processing of one or more neural network layers by the neural network hardware accelerator 110 (step 304). Specifically, the main memory 130 may determine the amount of available wide memory 210 capacity for each computing unit in the computing unit set 208 during the processing of one or more neural network layers. In various embodiments, the computing unit may have abundant wide memory 210, such as 8MB of memory, which is not fully utilized during neural network computation. Additionally, the wide memory 210 of the computing unit can provide higher bandwidth than the main memory 130.

[0044] In some implementations, the neural network processing system 100 identifies a subset of parameters for one or more neural network layers that consumes less memory than the difference between the memory capacity and a determined maximum memory capacity (step 306). In this illustration, the neural network processing system 100 identifies a subset of parameters for neural network computations in one or more computing units within the computing unit set 208 by splitting parameters based on the amount of one or more computing units and the available capacity of the wide memory 210 for one or more computing units. The neural network processing system 100 may also identify a subset of parameters corresponding to one or more specific neural network layers and send the subset of parameters to one or more computing units in the computing unit set 208 to perform neural network computations, such as convolution computations.

[0045] In the next step, the neural network hardware accelerator 110 stores the identified subset of parameters in at least one of one or more wide memories 210 of the hardware computing unit (step 308). In some embodiments, parameters are transferred from data memory 204 to one or more specific computing units of the computing unit set 208 via a ring bus 216. As previously described, the ring bus 216 may include a bitmap header indicating one or more computing units that need to consume data including computing unit instructions and model parameters. Upon receiving an instruction, one or more receiving computing units zero out (i.e., clear) the location data indicated in the bitmap header unique to the receiving computing unit before forwarding the parameter data to other adjacent computing units. In various embodiments, the bandwidth of the ring bus 216 corresponds to the bandwidth of the wide memories 210 disposed on the computing unit set 208.

[0046] In some implementations, while neural network computation instructions are being executed, the neural network processing system 100 may issue direct memory access operation instructions to one or more host computing units in the computing unit set 208 to transmit parameters stored in the host computing units to other receiver computing units via the ring bus 216 or mesh bus 214. In response, the receiver computing units may issue direct memory access operation instructions to copy the parameters transferred from the ring bus 216 to their wide memory 210. After storing the identified subset of parameters in the corresponding computing unit, the neural network hardware accelerator begins to execute neural network computation corresponding to that particular neural network layer (step 310).

[0047] In this illustration, the neural network can consist of multiple convolutional layers, which are composed of filters, nonlinear thresholding, and pooling operations. Once the desired number of convolutional layers are configured, the output is vectorized and passed through fully connected layers, subsequently producing the desired output. Convolutional layers typically have fewer parameters than fully connected layers. Fully connected layers produce the same output size and do not use bias for neural network computation. In various implementations, the neural network can include one or more fully connected layers that share the same amount of parameters used for computation at one or more computation units of the neural network hardware accelerator 110.

[0048] In some implementations, parameter cache classes can be created and deployed to support the parameter caching tasks described in the neural network hardware accelerator system 110. For example, given the available wide memory 210 in each computation unit, the parameter cache class can iterate through each neural network layer and determine the utilization of the wide memory 210 for parameter caching in each computation unit. The parameter cache is implemented in a first-come, first-served manner. In some other examples, the implementation of the parameter cache is fine-tuned for more complex tasks.

[0049] For a given neural network layer, it is preferable to allocate its model parameters within a single computational unit. This configuration reduces the instruction count when performing the corresponding neural network computation and simplifies the management of synchronization flags for the neural network hardware accelerator 110.

[0050] In the neural network processing system 100, compiler support is enabled for parameter caching. For example, a code generator can generate two programs: one to load parameters from the host interface 120 into the wide memory 210 of the computing unit, and another to execute an existing model of the neural network layers. Various functions can be used at runtime to obtain the corresponding binary files for these two programs. In some implementations, the required parameter cache is not sequentially ordered. For example, there may be five neural network layers with corresponding model parameters for neural network computation. The execution order of these five layers is:

[0051] A→B→C→D→E

[0052] If layers A, C, and E have cached parameters, then five direct memory access descriptors need to be generated to cache the parameters. In an alternative approach, parameter packing can be ordered as follows:

[0053] A→C→E→B→D

[0054] Only one direct memory access descriptor is needed for parameter packing in the sequence A→C→E, and another direct memory access descriptor is needed for parameter packing of non-cacheable parameters in the sequence B→D.

[0055] As previously stated, only one host computing unit is allowed to send parameters via ring bus 216. If two or more host computing units simultaneously send parameters to the same receiver computing unit via ring bus 216, the receiver computing unit may receive incorrect parameters, and data packets from inactive receiver computing units will float in ring bus 216, causing the system to hang.

[0056] Figure 4 This is an exemplary flowchart of a process for performing parameter caching for computation of fully connected neural network layers on a neural network hardware accelerator 110.

[0057] The neural network hardware accelerator 110 obtains data for one or more fully connected layers of a neural network to be deployed on one or more computing units of the neural network hardware accelerator 110, wherein each fully connected neural network layer shares a corresponding set of parameters. The host interface 120 receives instructions and model parameters for executing one or more fully connected neural network layers. The instructions and model parameters are transferred to main memory 130 and stored in instruction memory 206 and data memory 204, respectively.

[0058] The neural network processing system 100 partitions the set of parameters for computation of the fully connected layers on the neural network hardware accelerator 110 (step 402). In various embodiments, the neural network processing system 100 identifies a subset of the set of parameters for computation of the fully connected layers corresponding to the computation units in the set of computation units 208 by partitioning parameters based on the amount of the corresponding computation unit and the available wide memory 210 capacity of the corresponding computation unit. The neural network processing system 100 may have a global view of the parameters corresponding to the neural network layers and the model. It will determine the available capacity of the wide memory 210 in each computation unit and invoke a parameter caching method to allocate the wide memory 210 for parameter caching.

[0059] In some implementations, since no single computational unit is available to store the parameter set in its wide memory 210, the parameters can be partitioned at the Zout block granularity and distributed across various computational units. The final optimization can be achieved by maximizing parameter caching efficiency in terms of the capacity of the wide memory 210. For example, a particular model might have a fully connected layer with 1MB of parameters. However, the available wide memory 210 for each computational unit is less than 1MB, so the model parameters are not suitable for a single computational unit of the neural network hardware accelerator 110. As a result, the parameters of the fully connected layer are partitioned and cached across different computational units. In some implementations, the partitioned parameters are assigned to specific computational units to improve tensor computation efficiency, and unicasting of the partitioned parameters across the hardware computational unit set 208 is not required.

[0060] In the next step, the neural network processing system 100 allocates a subset of the fully connected layer parameters to one or more wide memories 210 of one or more computing units on the neural network hardware accelerator 110 (step 404).

[0061] The neural network hardware accelerator 110 sends the partitioned parameters one by one from the data memory 204 to one or more wide memories 210 of the identified one or more computing units via the ring bus 216 (step 406). In various embodiments, the fully connected neural network layer issues a RingInfeed instruction to transfer a subset of parameters from the data memory 204 to one or more specific computing units via the ring bus 216 for performing the corresponding neural network computation.

[0062] In the next step, the receiver computing units of computing unit set 208 receive a subset of parameters and write the parameters into one or more wide memories 210 of the computing units (step 408). The neural network processing system 100 may issue direct memory access operation instructions to one or more host computing units to store parameters from the host computing units to other adjacent receiver computing units via ring bus 216 or mesh bus 214. In response, the receiver computing units may issue direct memory access operation instructions to copy the parameters transferred from the host computing units to their wide memories 210 (step 410).

[0063] After storing the identified subset of parameters into the corresponding computational unit, the neural network hardware accelerator 110 begins performing neural network computations corresponding to one or more fully connected neural network layers. In various implementations, a particular neural network layer may have more than one model to be cached for parameters. Instructions indicating the list of models in descending priority can be sent to the main memory 130 of the hardware computational unit via the host interface 120 for execution. In response to the instructions, more important models can be processed earlier to cache their corresponding parameters in the wide memory 210 of one or more computational units.

[0064] Figure 5 This is an exemplary flowchart illustrating the process of performing parameter caching for computation of a convolutional neural network layer on a neural network hardware accelerator 110. A convolutional layer is a type of feedforward neural network layer suitable for manipulating and computing two-dimensional data such as images. Convolution is a linear operation involving the multiplication of model parameters—for example, an array of input data and a two-dimensional array of weights. The neural network hardware accelerator 110 delivers model parameters to one or more wide memories 210 of one or more computational units for computation of a particular convolutional neural network layer.

[0065] Upon receiving an instruction, main memory 130 specifies a set of model parameters for one or more convolutional neural network layers (step 502). The model parameters can be transferred from host interface 120 and stored in data memory 204.

[0066] The neural network processing system 100 can divide parameters into subsets based on their correspondence with one or more convolutional layers (step 504). In some implementations, the subset of parameters corresponding to a specific convolutional neural network layer is not suitable for the wide memory of the corresponding computation unit. As a result, the subset of parameters is further split and transferred to more computation units for convolution computation.

[0067] In the next step, the differentiated subset of parameters is transferred one by one from the data memory 204 to the wide memory 210 of the computing unit via the ring bus 216 (step 506). In some embodiments, the parameter cache is implemented in a simple first-come, first-served manner. In other embodiments, the parameter cache implementation algorithm is fine-tuned for more complex tasks.

[0068] The computing unit may copy the transmitted subset of parameters and write the parameter subset into its wide memory 210 (step 508). In some embodiments, since the subset of parameters is not suitable for the computing unit's wide memory 210, the computing unit receives parameters from an adjacent host computing unit and stores the parameters into its wide memory 210 to complete a portion of the convolution calculation.

[0069] After storing the differentiated subset parameters into the corresponding computation unit, the computation unit begins to execute neural network computation corresponding to one or more convolutional neural network layers upon receiving a tensor operation instruction (step 510).

[0070] Embodiments of the subject matter and functional operation described in this specification may be implemented as digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0071] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. Apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, apparatus may also include code that creates an execution environment for the computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0072] Computer programs (which may also be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and they can be deployed in any form, including as standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data, such as in one or more scripts in a markup language document; in a single file dedicated to said program; or in multiple coordinating files, such as in a file storing portions of one or more modules, subroutines, or code. A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communication network.

[0073] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and the devices can also be implemented as special-purpose logic circuitry.

[0074] Computers suitable for executing computer programs include, for example, those based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0075] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0076] To facilitate interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user—such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor—and a keyboard and pointing device, such as a mouse or trackball, for the user to send input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from devices used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0077] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a backend component as a data server; or a middleware component as an application server; or a frontend component as a client computer having, for example, a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification; or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium—e.g., a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0078] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0079] While this specification contains numerous details of specific implementation, these should not be construed as limiting any invention or potentially claimed scope, but rather as descriptions of features that may be specific to particular embodiments of the invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0080] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0081] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for parameter caching, comprising: Obtain data for one or more neural network layers to be deployed on a neural network hardware accelerator, each of the one or more neural network layers having a corresponding set of parameters, and the neural network hardware accelerator including a parameter memory and a plurality of hardware computing units, each of the plurality of hardware computing units being a processing element configured to perform neural network layer computations, and each hardware computing unit including one or more local memories having memory capacity. Store the one or more parameter sets in the parameter memory; The maximum amount of memory capacity to be used at any time during the processing of any one of the one or more neural network layers by the neural network hardware accelerator is determined by: determining the amount of available capacity of the local memory of each of the plurality of hardware computing units during the processing. Identify a subset of parameters of the one or more neural network layers, wherein the amount of memory consumed by the subset of parameters is less than the difference between the memory capacity and the determined maximum amount of the memory capacity; The identified subset of parameters is stored in at least one of the one or more local memories of the neural network hardware accelerator, wherein the identified subset of parameters is transferred from the parameter memory to the one or more local memories; as well as After storing the identified subset of parameters, the neural network hardware accelerator is made to perform processing on the one or more neural network layers that require the corresponding additional subset of parameters. Wherein, the one or more local memories of the neural network hardware accelerator provide higher bandwidth than the parameter memory storing the parameters of the neural network layers; and Each of the plurality of hardware computing units is further configured to perform an operation, the operation including: The processor of the corresponding hardware computing unit executes multiple instructions from the neural network hardware accelerator; and, selectively: Access one or more local memories of the corresponding hardware computing unit to retrieve the identified subset of parameters; and The identified subset of parameters is sent to one or more local memories of other hardware computing units in the plurality of hardware computing units of the neural network hardware accelerator via a ring bus; or the following is executed: A subset of identified parameters, received via the ring bus from one or more local memories of one or more other hardware computing units, is stored in one or more local memories of the corresponding hardware computing unit for use by the corresponding hardware computing unit to perform neural network computations using the identified subset of parameters.

2. The method according to claim 1, further comprising: Determine the storage location of the identified subset of parameters in the one or more local memories of the neural network hardware accelerator, and Each parameter is stored in the identified storage location used for that parameter.

3. The method according to claim 1, further comprising: The one or more neural network layers are sorted according to priority; as well as The parameter set of the highest-ranked layer in the neural network is identified as the parameter subset of the one or more neural network layers.

4. The method according to claim 1, wherein, The one or more neural network layers include fully connected layers, wherein the parameters of the fully connected layers are in an identified subset, wherein the accelerator is configured to distribute the processing of the fully connected layers across a first subset in the hardware computing unit, and wherein storing the identified subset includes: The corresponding partitions of the parameters of the fully connected layer are stored in one or more local memories of each hardware computing unit in the first subset.

5. The method according to claim 1, wherein, The one or more neural network layers include convolutional layers, wherein the parameters of the convolutional layers are in an identified subset, wherein the accelerator is configured to distribute the processing of the convolutional layers across a second subset in the hardware computing unit, and wherein storing the identified subset includes: The corresponding copies of the parameters of the convolutional layer are stored in the one or more local memories of each hardware computing unit in the second subset.

6. A system comprising a neural network hardware accelerator, one or more computers, and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers and the neural network hardware accelerator, to cause the one or more computers and the neural network hardware accelerator to perform operations, the operations including: Obtain data for one or more neural network layers to be deployed on a neural network hardware accelerator, each of the one or more neural network layers having a corresponding set of parameters, and the neural network hardware accelerator including a parameter memory and a plurality of hardware computing units, each of the plurality of hardware computing units being a processing element configured to perform neural network layer computations, and each hardware computing unit including one or more local memories having memory capacity. Store the one or more parameter sets in the parameter memory; The maximum amount of memory capacity to be used at any time during the processing of any one of the one or more neural network layers by the neural network hardware accelerator is determined by: determining the amount of available capacity of the local memory of each of the plurality of hardware computing units during the processing. Identify a subset of parameters of the one or more neural network layers, wherein the amount of memory consumed by the subset of parameters is less than the difference between the memory capacity and the determined maximum amount of the memory capacity; The identified subset of parameters is stored in at least one of the one or more local memories of the neural network hardware accelerator, wherein the identified subset of parameters is transferred from the parameter memory to the one or more local memories; as well as After storing the identified subset of parameters, the neural network hardware accelerator is made to perform processing on the one or more neural network layers that require the corresponding additional subset of parameters. Wherein, the one or more local memories of the neural network hardware accelerator provide higher bandwidth than the parameter memory storing the parameters of the neural network layers; and Each of the plurality of hardware computing units is further configured to perform an operation, the operation including: The processor of the corresponding hardware computing unit executes multiple instructions from the neural network hardware accelerator; and, selectively: Access one or more local memories of the corresponding hardware computing unit to retrieve the identified subset of parameters; and The identified subset of parameters is sent to one or more local memories of other hardware computing units in the plurality of hardware computing units of the neural network hardware accelerator via a ring bus; or the following is executed: A subset of identified parameters, received via the ring bus from one or more local memories of one or more other hardware computing units, is stored in one or more local memories of the corresponding hardware computing unit for use by the corresponding hardware computing unit to perform neural network computations using the identified subset of parameters.

7. The system according to claim 6, further comprising: Determine the storage location of the identified subset of parameters in the one or more local memories of the neural network hardware accelerator, and Each parameter is stored in the identified storage location used for that parameter.

8. The system according to claim 6, further comprising: The layers of the one or more neural networks are sorted according to priority; as well as The parameter set of the highest-ranked layer in the neural network is identified as the parameter subset of the one or more neural network layers.

9. The system according to claim 6, in, The one or more neural network layers include fully connected layers. The parameters of the fully connected layer are in the identified subset. The accelerator is configured to distribute the processing of the fully connected layer across a first subset of the hardware computing units, and The subset identified in the storage includes: The corresponding partitions of the parameters of the fully connected layer are stored in one or more local memories of each hardware computing unit in the first subset.

10. The system according to claim 6, in, The one or more neural network layers include convolutional layers. Wherein, the parameters of the convolutional layer are in the identified subset, The accelerator is configured to distribute the processing of the convolutional layer across a second subset of the hardware computing unit. The subset identified in the storage includes: The corresponding copies of the parameters of the convolutional layer are stored in the one or more local memories of each hardware computing unit in the second subset.

Images