Hardware implementations of neural networks

Abstract

Hardware implementations of neural networks and methods of processing data in such hardware implementations are disclosed. Input data for a plurality of layers of the network is processed in blocks to generate corresponding blocks of output data. The processing is performed through the plurality of layers in a depth direction, evaluating all of the layers for a given block before proceeding to the next block.

Description

Technical Field

[0001] This invention relates to a hardware implementation of a neural network. Background Technology

[0002] Deep neural networks (DNNs) are artificial neural networks that can be used in machine learning applications. In particular, DNNs can be used in signal processing applications, including image processing and computer vision applications.

[0003] DNNs have been implemented in applications where power resources are not a critical factor. Nevertheless, DNNs are used in many different technical fields where the hardware resources required for their implementation are limited by power consumption, processing power, or silicon area. Therefore, there is a need for hardware configured to implement DNNs (or at least a portion thereof) in an efficient manner, such as requiring less silicon area or less processing power during operation. Furthermore, DNNs can be configured in many different ways for various applications. Therefore, there is also a need for hardware that offers the flexibility to support diverse DNN configurations. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] A hardware implementation of a neural network and a method for processing data in such a hardware implementation are disclosed. Input data from multiple layers of the network is processed in blocks to generate corresponding blocks of output data. Processing is performed along the depth direction through multiple layers, thereby evaluating all layers in multiple layers for a given block before proceeding to the next block.

[0006] According to one aspect, a method for processing data in a hardware implementation of a neural network comprising multiple layers is disclosed, the method comprising:

[0007] Weight data representing the weights of multiple layers is provided in one or more first memory devices;

[0008] In one or more first memory devices, input data for a first layer of a plurality of layers is provided, the plurality of layers terminating at a generation p The final layer of each output dataset, each set including n One output data element;

[0009] Read a first subset of the input data representing a first block of input data from one or more first memory devices;

[0010] Read weight data from one or more first memory devices;

[0011] Process the weight data and a first subset of the input data to evaluate each of the multiple layers, thereby computing the first block of the output data for the first output dataset; and

[0012] Write the first block of output data to one or more first memory devices.

[0013] According to one example, a method for processing data in a hardware implementation of a neural network comprising multiple layers is disclosed, the method comprising:

[0014] Weight data representing the weights of multiple layers is provided in one or more first memory devices;

[0015] In one or more first memory devices, input data for a first layer of a plurality of layers is provided, the plurality of layers terminating at a generation p The final layer of each output dataset, each set including n There are 1 output data element, of which n >2 and p ≥1, and p It depends on the weighted data;

[0016] Read a first subset of the input data representing a first block of input data from one or more first memory devices;

[0017] Read weight data from one or more first memory devices;

[0018] Process the weight data and a first subset of the input data to evaluate each of the multiple layers, thereby computing the first block of output data for the first output dataset, wherein the first block of output data has m There are 1 output data element, of which m >1 and m < n ;as well as

[0019] Write the first block of output data to one or more first memory devices.

[0020] The inventors have recognized that it can be advantageous to process blocks of input data and thereby generate blocks of output data by sequentially evaluating several layers of a neural network. This depth-first processing of a subset of input data helps reduce the memory access overhead associated with writing the output of each layer of the network to or reading the output of each layer of the network from memory. Such memory access overhead can be very large in terms of time and / or energy costs, especially when the memory is off-chip memory.

[0021] When this depth-first approach is pushed to its logical limits, it becomes possible to compute only a single output data element at the end of multiple layers in each propagation (instead of blocks of output data as disclosed herein). However, the inventors of this invention have recognized that generating blocks of output data containing multiple output data elements in a single propagation can be advantageous. This can facilitate more efficient hardware implementations. For example, in a parallel implementation, multiple processing elements can be applied in parallel to generate the output data elements of the end layer. This enables efficient processing while leveraging potentially overlapping data dependencies to reduce memory bandwidth requirements.

[0022] The first subset of the input data (i.e., the first block) is processed as a whole through each individual layer before moving to the next layer. In other words, each layer is evaluated only once to compute the first block of the output data. This is what it means to process the weighted data and the first subset of the input data to compute the first block of the output data in a single propagation.

[0023] Input data can be multidimensional, including two or more dimensions. Input data may include 2D images or feature maps. Multiple layers can be multiple consecutive layers. Blocks of input data can be consecutive blocks of input data. Blocks of output data can be consecutive blocks of output data.

[0024] The first subset of the input data can be a subset selected on one, two, or more dimensions. For example, if the input data includes 2D image data or feature map data with x and y dimensions, the first subset can be selected on the x dimension, the y dimension, or both the x and y dimensions.

[0025] Here, a block of output data is the output data (from the final layer) corresponding to a block of input data. Therefore, a block of output data is a subset of the complete output data. A block of output data can have the same number of dimensions as the input data.

[0026] In some implementations, a first block of output data may be written to one or more first memory devices before processing of a second subset of the input data begins (or is completed). In some implementations, a first block of output data may be written to one or more first memory devices before reading a second subset from one or more first memory devices begins (or is completed). In other implementations (e.g., in a pipelined implementation), these conditions may be relaxed.

[0027] Evaluating a layer of a neural network means applying weight data to the input data of that layer to compute the output data. This may include multiplying the input values ​​in the input data by the corresponding weights in the weight data, and summing these multiplications to produce the output value in the output data. The output data of the current layer can be the input data of subsequent layers. At least one of the multiple layers is a convolutional layer.

[0028] It's important to note that the first layer in a multi-layer network doesn't necessarily have to be the first layer of the neural network. Similarly, the ending layer in a multi-layer network doesn't necessarily have to be the last layer. A multi-layer network can be a subset of the layers in a neural network. However, in some cases (especially for networks with a small number of layers), a multi-layer network can include all the layers of the neural network.

[0029] The method may also include providing one or more processing elements configured to process weights and a first subset of input data to evaluate each of a plurality of layers. The multiple processing elements can operate in parallel to evaluate a given layer of the neural network. This includes parallel operation to evaluate the output layer. The neural network may be a deep neural network.

[0030] p Each output dataset in the set of output datasets can represent an output data plane, such as a feature map. The number of output datasets can depend on the weight data, as the weight data includes a set of weights used to define each output dataset. This set of weights can form a filter kernel, and each filter kernel, for example, defines an output feature map.

[0031] The first block of output data may include multiple output data elements extending in one or more dimensions, and the first subset of input data may include multiple input data elements extending in one or more dimensions, wherein each output data sub-element depends on a corresponding subgroup of input data elements, and each subgroup has at least one input data element common to at least one other subgroup.

[0032] Therefore, the output data element depends on a subgroup of input data elements that overlap in one or more dimensions. Here, "depends on" a subgroup of input data elements means that the input data elements in that subgroup (and only those input data elements) are used in the computation of the output data element.

[0033] "Overlap" between two subgroups refers to the common input data elements of the two subgroups. Not all subgroups must overlap. For example, only adjacent output data elements may depend on overlapping subgroups. It should also be noted that overlap is preferably partial, whereby the subgroups are distinct subgroups that share some input data elements, but not all of their input data elements are common.

[0034] The method may further include providing one or more second memory devices, wherein the cost of accessing the one or more second memory devices is less than the cost of accessing one or more first memory devices, wherein processing a first subset of weight data and input data includes: processing the first subset of weight data and input data to evaluate a first layer among a plurality of layers, thereby calculating output data of the first layer; writing the output data of the first layer to the one or more second memory devices; and for each subsequent layer among the plurality of layers: reading the output data of the previous layer from the one or more second memory devices; and processing the weight data and the output data of the previous layer to evaluate the current layer among the plurality of layers, thereby calculating the output data of the current layer.

[0035] The method may also include, for each subsequent layer after the first layer, writing the output data of the current layer to one or more second memory devices (except, optionally, for the end layer, the output buffer may be configured to write the output data to one or more first memory devices).

[0036] One or more first memory devices may be off-chip memory devices, such as synchronous dynamic random access memory (SDRAM). One or more second memory devices may be on-chip memory, such as on-chip RAM. The cost of accessing the memory devices may be energy cost and / or time cost.

[0037] The method may further include writing the weight data to one or more second memory devices after reading the weight data from one or more first memory devices. Optionally, the method includes reading the weight data of a layer from one or more second memory devices before evaluating each layer.

[0038] For at least one of multiple layers, the method may further include: when computing output data for at least one layer: identifying disposable portions of the output data for at least one layer, which may be deleted after computing a first block of output data for the final layer; and identifying non-disposable portions of the output data for at least one layer, which should be retained after computing a first block of output data for the final layer for use in computing at least one other block of output data for the final layer. Optionally, the method includes: when writing output data for at least one layer to one or more second memory devices: writing disposable portions to a first segment of one or more second memory devices; and writing non-disposable portions to a second different segment of one or more second memory devices.

[0039] The one-time write may include a one-time portion that rewrites the output data of a previous layer that was previously written to a first segment of one or more second memory devices. Here, "previous" refers to the previous part during the calculation of the first output block of the final layer.

[0040] In this way, the one-time portion of the output data can be stored for a short period in the first segment of memory, which is frequently rewritten. The non-one-time portion of the output data can be stored for a longer period in the second segment of memory, which is rewritten at a lower frequency. This facilitates the efficient use of memory resources.

[0041] When evaluating subsequent layers among multiple layers, at least a portion of the one-time portion of the output data can be rewritten. This one-time portion can be rewritten from the one-time portion of the output data of the subsequent layers.

[0042] Double buffering can be performed on a second segment of one or more second memory devices. This allows non-one-time portions of a previous block to be read from one memory location (a buffer) and allows non-one-time portions of the current block to be written to a second memory location (a second buffer) without rewriting data that is still needed in the computation. To process the next block, the memory location used for reading and writing is switched.

[0043] The method may further include, when evaluating subsequent layers of multiple layers: reading a one-time portion from a first segment of one or more second memory devices; reading a non-one-time portion from a second segment of one or more second memory devices; and using the one-time portion and the non-one-time portion in the calculation of the output of the subsequent layer.

[0044] The method may further include: reading a second subset of input data representing a second block of input data from one or more first memory devices; processing weight data and the second subset of input data to evaluate each of a plurality of layers, thereby computing a second block of output data for a first output dataset; and writing the second block of output data to one or more first memory devices. The second block may be adjacent to the first block. The second block may extend along one or more dimensions of the same as the first block.

[0045] The method may further include: reading a second subset of input data representing a second block of input data from one or more first memory devices; reading a non-one-time portion of output data from at least one of a plurality of layers from one or more second memory devices, the non-one-time portion having previously been written to one or more second memory devices during the computation of a first block of output data for a first output dataset; processing weight data, the non-one-time portion, and the second subset of input data to evaluate the plurality of layers, thereby computation of a second block of output data for the first output dataset; and writing the second block of output data to one or more first memory devices.

[0046] This method can be performed block by block until all blocks of the first output dataset have been computed. If the final layer generates more than one output dataset, the corresponding blocks of the output data for the other output datasets can be computed in the same steps as computed for each block of the output data of the first output dataset.

[0047] It also provides a hardware implementation of a neural network with multiple layers, which includes:

[0048] One or more first memory devices, the one or more first memory devices being configured to store:

[0049] Weight data representing the weights of multiple layers; and

[0050] The input data for the first layer of a plurality of layers, wherein the plurality of layers terminate at generation p The final layer of each output dataset, each set including n There are 1 output data element, of which n >2 and p ≥1, and p Depending on the weighted data,

[0051] An input buffer, configured to receive a first subset of the input data representing a first block of input data.

[0052] A coefficient buffer, configured to obtain weighted data, and

[0053] One or more processing elements are configured to process weight data and a first subset of input data to evaluate each of a plurality of layers, thereby computing a first block of output data for a first output dataset, wherein the first block of output data has m There are 1 output data element, of which m >1 and m < n ;as well as

[0054] An output buffer, configured to output the first block of data. At least one of the multiple layers is a convolutional layer.

[0055] In some examples, the input buffer may be configured to read a first subset of input data directly from one or more first memory devices. Alternatively, one or more second memory devices may be configured to receive a first subset of input data from one or more first memory devices, and the input buffer may be configured to read that data from one or more second memory devices. Similarly, in some examples, the coefficient buffer may be configured to read weight data directly from one or more first memory devices. Alternatively, one or more second memory devices may be configured to receive weight data from one or more first memory devices, and the coefficient buffer may be configured to read weight data from one or more second memory devices. Likewise, in some examples, the output buffer may be configured to write a first block of output data directly to one or more first memory devices. Alternatively, the output buffer may be configured to write a first block of output data to one or more second memory devices, and one or more first memory devices may be configured to receive a first block of output data from one or more second memory devices. All combinations of these alternatives are disclosed accordingly.

[0056] The hardware implementation may further include one or more second memory devices, wherein the cost of accessing the one or more second memory devices is less than the cost of accessing the one or more first memory devices, wherein one or more processing elements are configured to process weight data and a first subset of input data to evaluate a first layer among a plurality of layers, thereby calculating output data of the first layer; an output buffer is configured to write the output data of the first layer to the one or more second memory devices; and for each subsequent layer in the plurality of layers: an input buffer is configured to read output data of the previous layer from the one or more second memory devices; and one or more processing elements are configured to process weight data and output data of the previous layer to evaluate the current layer among a plurality of layers, thereby calculating output data of the current layer.

[0057] For each subsequent layer after the first layer, the output buffer can be configured to write the output data of the current layer to one or more second memory devices (except optionally, for the end layer, the output buffer can be configured to write the output data to one or more first memory devices).

[0058] One or more second memory devices may be configured to obtain weight data from one or more first memory devices, and a coefficient buffer may be configured to read the weight data of a layer from one or more second memory devices before each layer is evaluated.

[0059] When the output data of at least one of a plurality of layers has been calculated, the output buffer can be configured to: identify a one-time portion of the output data of at least one layer, which can be deleted after the first block of the output data of the final layer has been calculated; identify a non-one-time portion of the output data of at least one layer, which should be retained after the first block of the output data of the final layer for use in at least one other block of the output data of the final layer; write the one-time portion to a first segment of one or more second memory devices; and write the non-one-time portion to a second different segment of one or more second memory devices.

[0060] When evaluating subsequent layers among multiple layers, the output buffer can be configured to rewrite at least a portion of a one-time portion of the output data. The output buffer can be configured to rewrite a one-time portion with the one-time portion of the output data from the subsequent layer.

[0061] Double buffering can be performed on the second segment of one or more second memory devices.

[0062] When evaluating a subsequent layer among multiple layers: the input buffer can be configured to read a one-time portion from a first segment of one or more second memory devices; the input buffer can be configured to read a non-one-time portion from a second segment of one or more second memory devices; and one or more processing elements can be configured to use the one-time portion and the non-one-time portion in the calculation of the output of the subsequent layer.

[0063] The input buffer can be configured to obtain a second subset of the input data representing a second block of the input data; one or more processing elements can be configured to process the weight data and the second subset of the input data to evaluate each of the multiple layers, thereby computing a second block of the output data of the first output dataset; and the output buffer can be configured to output the second block of the output data.

[0064] The input buffer can be configured to obtain a second subset of the input data representing a second block of input data; the input buffer can be configured to read a non-one-time portion of the output data from at least one of a plurality of layers from one or more second memory devices, the non-one-time portion having previously been written to one or more second memory devices during the computation of a first block of the output data of a first output dataset; one or more processing elements can be configured to process the weight data, the non-one-time portion, and the second subset of the input data to evaluate the plurality of layers, thereby computation of a second block of the output data of the first output dataset; and the output buffer can be configured to output the second block of the output data.

[0065] The weight data may optionally include one or more weight sets for each of the multiple layers, and p depends on the number of weight sets for the final layer.

[0066] The following are also disclosed: a processing system configured to perform the methods outlined above; and a processing system comprising the hardware implementations outlined above. The processing system may be a graphics processing system or an artificial intelligence accelerator system. The processing system may be implemented in hardware on an integrated circuit.

[0067] A method for manufacturing the hardware implementation or processing system described above using an integrated circuit manufacturing system is also provided.

[0068] A method for manufacturing the processing system as described above using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the processing system; and manufacturing the processing system based on the circuit layout description using an integrated circuit generation system.

[0069] A computer-readable code is also provided, configured to enable the methods outlined above to be executed when the code is run; and a computer-readable storage medium on which the computer-readable code is encoded. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0070] It also provides an integrated circuit definition dataset, which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the hardware implementations or processing systems as outlined above.

[0071] A computer-readable storage medium is also provided, on which a computer-readable description of a processing system as outlined above is stored, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the processing system.

[0072] A computer-readable storage medium is also provided, on which a computer-readable description of a processing system as described above is stored, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to perform the following operations: process the computer-readable description of the processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the processing system; and manufacture the processing system using an integrated circuit generation system based on the circuit layout description.

[0073] An integrated circuit manufacturing system is also provided, which is configured to manufacture the processing system as described above.

[0074] An integrated circuit manufacturing system is also provided, comprising: a non-transitory computer-readable storage medium storing a computer-readable description of a processing system as outlined above; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the processing system; and an integrated circuit generation system configured to manufacture the processing system according to the circuit layout description.

[0075] A layout processing system can be configured to determine the location information of logic components of a circuit derived from an integrated circuit description in order to generate a circuit layout description of the integrated circuit that embodies the processing system.

[0076] Hardware implementations or processing systems can be embodied in hardware on an integrated circuit. A method for manufacturing hardware implementations or processing systems at an integrated circuit manufacturing system can be provided. An integrated circuit definition dataset can be provided, which, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware implementations or processing systems. A non-transitory computer-readable storage medium can be provided, storing a computer-readable description of a hardware implementation or processing system, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the hardware implementation or processing system.

[0077] An integrated circuit manufacturing system may be provided, comprising: a non-transitory computer-readable storage medium storing a computer-readable description of a hardware implementation or processing system; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the hardware implementation or processing system; and an integrated circuit generation system configured to manufacture the hardware implementation or processing system based on the circuit layout description.

[0078] Computer program code for performing any of the methods described herein may be provided. A non-transitory computer-readable storage medium may be provided having computer-readable instructions stored thereon, which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0079] As will be apparent to those skilled in the art, the above features can be appropriately combined, and can be combined with any aspect of the examples described herein. Attached Figure Description

[0080] The example will now be described in detail with reference to the accompanying drawings, in which:

[0081] Figure 1This is a schematic diagram of an exemplary deep neural network (DNN);

[0082] Figure 2 This is a schematic diagram of exemplary data in a DNN;

[0083] Figure 3 This is a block diagram of an exemplary hardware implementation of a convolutional layer in a DNN;

[0084] Figure 4 yes Figure 3 A block diagram of an exemplary convolutional engine;

[0085] Figure 5 This is a schematic diagram illustrating the data flow according to an exemplary hardware implementation.

[0086] Figure 6 This shows the data dependencies across multiple layers of the DNN for the first block of data;

[0087] Figure 7 The second block of data is shown. Figure 6 Data dependencies within multiple layers;

[0088] Figure 8 It is a concept map of overlapping data concepts;

[0089] Figure 9 An exemplary allocation of memory associated with the hardware implementation of a DNN is illustrated schematically;

[0090] Figure 10 A DNN with four layers and no branches is shown;

[0091] Figure 11A An exemplary grouping of layers in a DNN with four layers having branches is shown;

[0092] Figure 11B Showing the target Figure 11A An exemplary grouping of alternatives to DNNs;

[0093] Figure 12 The computer system in which the artificial intelligence accelerator system is implemented is shown; and

[0094] Figure 13 An integrated circuit manufacturing system for generating integrated circuits that embody an artificial intelligence accelerator system is shown.

[0095] The accompanying drawings illustrate various examples. Those skilled in the art will understand that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the drawings represent one example of a boundary. In some examples, it may be that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Implementation

[0096] The following description is given by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art. Embodiments are now described by way of example only.

[0097] A deep neural network (DNN) is an artificial neural network that consists of multiple interconnected layers that enable the DNN to perform signal processing tasks, including but not limited to computer vision tasks. Figure 1 An exemplary DNN 100 is shown, comprising multiple layers 102-1, 102-2, and 102-3. Each layer 102-1, 102-2, and 102-3 receives input data and processes that input data according to the layer to produce output data. The output data is either provided to that layer as input data to another layer or output as the final output data of the DNN. For example, in Figure 1 In the DNN 100, the first layer 102-1 receives the raw input data 104 from the DNN 100 and processes the input data to produce output data. The output data of the first layer 102-1 becomes the input data of the second layer 102-2, which processes the input data to produce output data. The output data of the second layer 102-2 becomes the input data of the third layer 102-3, which processes the input data to produce output data. The output data of the third layer 102-3 is output as the output data 106 of the DNN.

[0098] The processing performed on the input data of a layer depends on the type of layer. For example, each layer of a DNN can be one of several different types. Exemplary DNN layer types include, but are not limited to, convolutional layers, activation layers, normalization layers, pooling layers, and fully connected layers. It will be apparent to those skilled in the art that these are exemplary DNN layer types and that this is not an exhaustive list, and that other DNN layer types may exist.

[0099] Convolutional layers are configured to convolve the input data using weights associated with that layer. Specifically, each convolutional layer is associated with multiple weights w1… w iRelatedly, these weights can also be referred to as filter weights or coefficients. Weights can be grouped to form or define one or more filters or kernels.

[0100] refer to Figure 2 This figure illustrates an example of the data format 200 used in a DNN. (As shown in...) Figure 2 As can be seen, the data 200 used in a DNN can be arranged into p data planes, where each plane has a size of x×y. A DNN can include one or more convolutional layers, where each convolutional layer is associated with multiple filters, each filter including multiple weights. Each filter has a size of m×n×p (i.e., each filter includes a set of m×n×p weights w), and is applied to the input data according to convolution operations across several steps (called strides) s and t, as... Figure 2 As shown. The group of data elements to which filters are applied at each stride is called a window. Each filter produces an output plane. The number of filters and the number of weights for each filter can vary between convolutional layers. Convolutional Neural Networks (CNNs) are an efficient method for image recognition and classification; they are a special type of DNN and generally consist of multiple convolutional layers.

[0101] Hardware implementations of convolutional layers may include hardware modules or blocks (referred to herein as "processing elements") configured to compute the sum of products between all or a portion of the weights forming a filter and all or a portion of the input data values ​​forming a window (referred to as filter window computation). Since a large number of such filter window computations are typically performed, some hardware implementations may include multiple such processing elements, allowing more than one filter window computation to be performed in parallel. Preparing each processing element to perform a filter window computation involves reading the appropriate input data and weights for each filter window computation from memory and providing them to one of the processing elements. Typically, a large amount of data needs to be transferred from memory to the processing element. Failure to do this efficiently can result in high memory bandwidth requirements and high power consumption for providing input data and weights to the processing element. This is especially true when the memory is "off-chip" memory, i.e., implemented in a different integrated circuit or semiconductor die than the processing element.

[0102] Therefore, this paper describes hardware implementations of neural networks and methods for processing data in such hardware implementations, wherein input data and weights are provided to the processing element in an efficient manner to reduce the memory bandwidth and power consumption required to provide the input data and weights to the processing element. Specifically, blocks of data are processed in a depth-oriented manner, passing through multiple layers of the neural network. Intermediate outputs of layers within multiple layers can be stored in one or more second memory devices. The size of the one or more second memory devices may be smaller than the size of the one or more first memory devices used for bulk data storage, but can be accessed in a faster and / or more energy-efficient manner. In one example, the one or more second memory devices are disposed on a chip (i.e., in the same integrated circuit or semiconductor die as the processing element).

[0103] Now for reference Figure 3 This figure illustrates an exemplary hardware implementation 300 of a convolutional layer in a DNN. Hardware implementation 300 includes multiple convolutional engines 302, multiple accumulators 304, an accumulation buffer 306, a coefficient buffer 308, and an input buffer 310. Each convolutional engine 302, along with its corresponding accumulator 304 and its share of resources in the accumulation buffer 306, represents a processing element 318. The hardware implementation also includes double data rate (DDR) SDRAM 312 as off-chip memory; on-chip memory (OCM) 314; and an output buffer 316.

[0104] Each convolutional engine 302 includes hardware logic configured to receive a set of weights (e.g., {w1 … w8}) representing all or a portion of a filter and a set of input data values ​​(e.g., {d1 … d8}) representing all or a portion of a window, and to perform multiplication-accumulation calculations on the received weights and input data values, such as Figure 4 As shown. In some examples, such as Figure 4 As shown, each convolutional engine 302 may include multiple multipliers 402, each multiplier being configured to multiply the weights (w... i ) and the corresponding input data value (d) i The multipliers 402 and 404 are multiplied to produce a multiplication output value. Following multiplier 402 are multiple adders 404 forming an adder tree to calculate the sum of the multiplication outputs. Figure 4 In the example, the convolution engine 302 includes eight multipliers 402, but in other examples, there may be more or fewer multipliers. For example, in some cases, there may be 128 multipliers. Typically, if there are Z multipliers 402, the adder tree includes Z-1 adders 404.

[0105] In some cases, multiplication-accumulation computations are pipelined. For example, multiplier 402 and adder 404 may be divided into multiple pipeline stages, with a register stage (not shown) preceding the first pipeline stage and between each pair of pipeline stages. For example, the multiplier may form the first pipeline stage, and the adder may be divided into layers, with subsequent pipeline stages comprising one or more adder layers.

[0106] Figure 3 An exemplary hardware implementation 300 includes four convolution engines 302; however, it will be apparent to those skilled in the art that the methods and principles described herein are applicable to hardware implementations with any number of convolution engines.

[0107] Because a convolution engine may require more than one hardware propagation process to generate the complete filter result (e.g., because the convolution engine may only receive and process a portion of the filter weights and / or a portion of the input data values ​​of the window in a loop), the hardware implementation may include multiple accumulators 304. Each accumulator 304 receives the output of a convolution engine 302 and adds that output to a previous convolution engine output associated with the same filter. Since the convolution engine may not generate or produce outputs associated with the same filter in consecutive loops, partial results of one or more filters may be stored in an accumulation buffer 306, which can then provide the appropriate partial results to the accumulator in each loop. In some examples, the accumulation buffer 306 may be able to store partial results associated with 128 different filters.

[0108] The coefficient buffer 308 includes: a memory (not shown) for storing multiple weights associated with the convolutional layers (or fully connected layers) of the DNN; and hardware logic (not shown) for providing the weights to the convolutional engine 302 for processing in a predetermined order across multiple loops. The multiple weights may include all weights associated with the convolutional layers, or only a portion of the weights associated with the convolutional layers. Although the coefficient buffer 308 is shown as a single module, it can be implemented, for example, by multiple coefficient buffers, each forming a memory volume.

[0109] The input buffer 310 includes: a memory (not shown) for storing multiple input data values ​​associated with convolutional layers (or fully connected layers) of the DNN; and hardware logic (not shown) for providing the input data values ​​to the convolutional engine 302 for processing in a predetermined order across multiple loops. The multiple input data values ​​may include all input data values ​​associated with the convolutional layers, or only a subset of the input data values ​​associated with the convolutional layers. Although in Figure 3The input buffer 310 is shown as a single module, but for example, the input buffer 310 may be implemented by multiple input buffers, each forming a memory bank.

[0110] DDR 312 is coupled to on-chip memory 314 for providing weight data to on-chip memory 314. DDR 312 is also coupled to input buffer 310 for providing blocks of input data to input buffer 310. On-chip memory 314 is coupled to coefficient buffer 308 for providing weight data to coefficient buffer 308. On-chip memory 314 is also coupled to input buffer 310 for providing intermediate output data (including input data for subsequent layers) to input buffer 310. On-chip memory 314 is also coupled to DDR 312 for providing blocks of output data from the end layer of multiple layers to DDR 312. Accumulation buffer 306 is coupled to output buffer 316 to allow the output buffer to receive intermediate output data from multiple layers, as well as output data from the end layer. Output buffer 316 is coupled to on-chip memory 314 for providing intermediate output data and output data from the end layer to on-chip memory 314.

[0111] exist Figure 3 The examples illustrate various connections; however, in some implementations, some or all of them may be provided by one or more shared bus connections. It should also be understood that other connections may be provided as... Figure 3 Alternatives or supplements to the connections shown. For example, output buffer 316 can be coupled to DDR 312 to provide output data directly to DDR 312. As a further example, DDR 312 can be coupled to coefficient buffer 308 to provide weight data directly to coefficient buffer 308. Similarly, in some cases, not... Figure 3 All the connections shown are necessary. For example, the DDR 312 does not always need to be coupled to the input buffer 310, which can instead obtain input data from the DDR 312 via the on-chip memory 314.

[0112] Figure 5 This is a simplified diagram to better illustrate the situation based on similar... Figure 3 A data stream is an example of a hardware implementation. Figure 5The diagram illustrates multiple layers of a neural network (consisting of two layers 502-1 and 502-2). These can be any two consecutive layers of the network. It should also be noted that multiple layers are not limited to two layers. The diagram shows input data 504 to the first layer 502-1 of the multiple layers. Intermediate output data 506 of the first layer 502-1 forms the input data to the second layer 502-2 (the ending layer in this example of two layers). The diagram also shows the output data 508 of the second layer 502-2 (the ending layer). DDR 312 is a first memory device with relatively high read / write costs. On-chip memory 314 is a second memory device with relatively low read / write costs. For example, reading / writing a given amount of data from DDR consumes more energy than reading / writing from on-chip memory 314. It should be understood that DDR memory is merely one example of a high-capacity memory storage device, and other memory / storage technologies, such as flash memory, can also be used.

[0113] according to Figure 3 and Figure 5 For example, DDR 312 stores input data for the first layer 502-1 of multiple layers 502-1, 502-2. DDR 312 also stores weight data representing the weights of the multiple layers 502-1, 502-2. In this example of the invention, the weight data is transferred from DDR 312 to on-chip memory 314. Coefficient buffer 308 reads the weight data from on-chip memory 314 and provides the weight data to processing element 318. Input buffer 310 reads a first subset of input data representing a first block of input data 504 from DDR 312. This first subset of input data 504 is... Figure 5 The data is shown in gray shading and may include one or more data planes. Input buffer 310 provides a first subset of input data 504 to processing element 318. Processing element 318 processes the weight data and the first subset of input data 504 to evaluate each of the multiple layers, thereby computing a first block 508 of output data for the final layer. The first block 508 of output data is... Figure 5 Shown in gray shades, it may include one or more data planes. Output buffer 316 can write a first block of output data 508 to on-chip memory 314. The first block of output data 508 can be transferred from the on-chip memory to DDR 312. Alternatively, the output buffer can write output data directly to DDR memory 312 (e.g., ...). Figure 3 and Figure 5 (As shown by the dashed arrow in the image).

[0114] like Figure 5As shown, the data input to and output from each layer is arranged in one or more planes (also referred to in this paper as the input dataset and output dataset). Figure 5 In the example shown, input data 504 includes p in =3 data planes. The intermediate output data 506 of the first layer 502-1 includes p1=3 data planes. The output data 508 of the second layer 502-2 includes p=2 data planes. The number of planes depends on the weight data. Specifically, the weights constituting each filter take a predetermined number of input planes as input data, and each filter outputs a plane of output data. Therefore, the number of data planes generated by a given layer is equal to the number of filters applied to that layer. Each data plane can represent a feature map. In Figure 5 In the example, the three planes of input data 504 can represent three feature maps generated by an earlier layer of the neural network (if the first layer 502-1 in the figure is not the first layer of the neural network). Alternatively (if the first layer 502-1 is the first layer of the neural network), the three planes of input data 504 can represent three input data planes. For example, input data with multiple planes often occurs in image processing tasks. Such input planes can include, but are not limited to: red, green, and blue (RGB) planes; or hue, saturation, and lightness (HSV) planes. Input planes can also include depth data as one of the input planes.

[0115] To compute the first block of output data, processing element 318 must evaluate each of the multiple layers (as described above). Specifically, coefficient buffer 308 reads the weight data of the first layer 502-1 from on-chip memory 314. Processing element 318 processes the weight data of the first layer 502-1 and a first subset of the input data 504 to evaluate the first layer 502-1 among the multiple layers, thereby computing the output data 506 of the first layer. This is a block of output data of the first layer 502-1 corresponding to the first block of input data 504. In other words, it is a subset of the complete set of output data of the neural network of the first layer 502-1. This block / subset is in Figure 5Shown in gray shading. Output buffer 316 writes a block of output data from layer 502-1 to on-chip memory 314. This represents intermediate output data that will be needed as input data to the next layer (layer 2 502-2). To evaluate the next layer (layer 2 502-2), input buffer 310 reads a block of output data 506 from layer 1 502-1 stored in on-chip memory 314 by output buffer 316. Coefficient buffer 308 reads weight data for layer 2 502-2 from on-chip memory 314. Processing element 318 processes the weight data for layer 2 502-2 and the block of output data 506 from the previous (first) layer to evaluate layer 502-2, thereby calculating the output data for layer 2 502-2. For any subsequent inner layer of multiple layers, this process continues—each time intermediate output data is stored in on-chip memory 314 and read to provide input data to the next layer. This continues until the last layer is reached. Figure 5 In the example, the second layer 502-2 is the end layer. Therefore, the output of the second layer 502-2 is the first block of output data 508. As explained above, the output buffer 316 can write the first block of output data 508 into the on-chip memory 314. The first block of output data 508 can then be transferred from the on-chip memory to DDR 312. Alternatively, the output buffer 316 can write the output data directly into DDR 312 (dashed arrow).

[0116] Processing a subset or block of input data through multiple layers of the neural network allows for efficient utilization of the processing resources of processing element 318. Writing intermediate output data to and reading it from on-chip memory 314 to process the next layer reduces the need for reading and writing data from off-chip DDR 312, thus contributing to lower power consumption. It's important to note that on-chip memory 314 may be significantly smaller than off-chip DDR 312 because during each propagation, on-chip memory 314 only needs to store data associated with the current block of input data, the current block of output data, and the associated blocks of intermediate output data for inner layers. This contrasts sharply with conventional methods for evaluating neural network layers, which process each layer as a whole independently, requiring memory capable of storing all input and output data for a layer at any given time. This necessitates a much larger memory, often impractical to implement as on-chip memory. Therefore, conventional practices require the use of off-chip memory to store intermediate output data. Since off-chip read and write operations typically consume more energy than on-chip read and write operations, reducing the amount of data written to off-chip memory significantly saves power.

[0117] The size of the block to be processed in each propagation can be selected based on the available storage capacity of the on-chip memory 314, the number of layers in multiple layers, and the structure of the neural network. An exemplary method of memory management in the on-chip memory 314 will be described below. First, it is useful to consider how the blocks of output data from multiple layers in a neural network depend on the intermediate output data of previous layers and the input data of the first layer. (Reference) Figure 6 This is understandable. The figure illustrates the input data of three layers (not explicitly shown) of a neural network; the output data of the first layer (labeled layer 0 here); the output data of the second layer (labeled layer 1); and the output data of the final layer. For simplicity, the output data of the final layer can conveniently be viewed as a single feature map created by a single filter. Similarly, the other layers can be viewed as convolutional layers with single filters, thus creating single output feature maps, as shown in a simplified form in the figure. Of course, the scope of this disclosure is not limited to this approach; these are merely simplifications for ease of understanding. At each layer, a single filter can be applied to multiple input channels (feature maps), or multiple filters can be applied to one or more input channels to process and / or generate multiple data planes. The methods and hardware implementations according to this disclosure can be used equivalently in such scenarios.

[0118] Starting with the first block of output data (patch 1) in the final layer, we can trace back through the network to find the intermediate and input data that this output block depends on. There is a block of data (patch 1) in the layer 1 output that corresponds to patch 1 in the final layer; in this sense, patch 1 in layer 1 is needed to compute patch 1 in the final layer. Assume the filter kernel size is greater than 1. Due to this kernel size, patch 1 in the layer 1 output is larger than patch 1 in the final layer output. In other words, patch 1 in the final layer output depends on some additional data in the layer 1 output. For example, for a 3x3 kernel and a stride of 1, an extra row of output data is needed in the layer 1 output. Figure 6 The dashed lines in the diagram indicate this expansion of the required data volume. In this figure, each tile is separated by a solid line.

[0119] Next, consider which data in the layer 0 output is necessary to compute tile 1 in the layer 1 output. For diversity, we will assume that the stride of layer 1 is greater than 1. This means that the layer 1 output is smaller than the layer 0 output. There is a tile in layer 0, namely tile 1, which corresponds to tile 1 in layer 1. Since the kernel size is also greater than 1, the amount of data required at the boundaries increases further. Therefore, tile 1 in the layer 0 output is larger than tile 1 in the layer 1 output.

[0120] The same situation occurs when traversing back from layer 0 to the input data. The input data contains a block, namely tile 1, corresponding to tile 1 in layer 0. Assume the stride in layer 0 is 1; therefore, the size of the input data is approximately the same as the size of the layer 0 output (depending on padding). Similarly, since the kernel size is greater than 1, additional data is needed at the boundaries. Therefore, tile 1 in the input data is also larger than tile 1 in the layer 0 output. Tile 1 in the input data forms the first subset (tile 1) of the input data. Tile 1 in the output data of the final layer forms the output block.

[0121] As can be inferred from the above, the total amount of data required for the first tile increases sequentially from the end tile to the input tile. This depends specifically on the parameters of all intermediate layers (kernel size, stride, and dilation). Figure 6 The diagram also illustrates "overlap." This is data that can be reused in subsequent propagation processes, i.e., data reused when computing subsequent tiles. Overlap data is indicated by horizontal dashed lines in the data of each layer. Overlap 1 in the output of layer 1 is a portion of tile 1 and can be (repeatedly) used to compute the subsequent tile (tile 2) in the output of the final layer. Overlap 1 in the output of layer 0 is a portion of tile 1 and can be (repeatedly) used to compute tile 2 in the output of layer 1. The size of the overlap depends only on the parameters of the next layer (kernel size, dilation). For example, if stride = 1 and dilation = 1, the size of the overlap (the number of overlapping rows) is the kernel size minus 1.

[0122] Immediately afterwards Figure 6 , Figure 7 The data dependencies for the second block are shown. The second block (block 2) of the output data of the final layer is adjacent to the first block (block 1). As with block 1, block 2 depends on the amount of intermediate data (ultimately input data) added as the layers are traversed backward. However, some of these data dependencies are the same as those of block 1 in the output data of the final layer. Specifically, when evaluating block 1 (block 1) in layer 1, overlap 1 in the output of layer 1 has been calculated. The only additional data that needs to be calculated is the block in layer 1, i.e., block 2. Similarly, moving backward to the output of layer 0, the only additional data that needs to be calculated is the block in layer 0, i.e., block 2. Finally, moving backward to the input data, the only additional input data that needs to be read from memory is block 2 (block 2). Figure 6 As shown, Figure 7Each tile in the graph is separated by a solid line. Overlapping data is separated by dashed lines. Dashed lines indicate the highest limit of a data dependency (i.e., the first row of data required to compute a tile in the next layer). Sloping dashed lines indicate the lowest limit of a data dependency (i.e., the last row of data required to compute a tile in the next layer). Thus, for example, tile 2 in the output of layer 1 depends on overlap 1 and tile 2 in the output of layer 0. Tile 2 contains overlap 2, which will be reused in the next propagation to compute tile 3 in subsequent layers, and so on. Note that if the overlap data from the first propagation is preserved, the amount of “fresh” data required for tile 2 does not increase when traversing these layers backward (except where a stride greater than 1 indicates). The same applies to all subsequent propagations (tiles) after tile 2. In other words, for all propagations after the first propagation (first tile), a smaller amount of input data needs to be retrieved from memory, and a smaller amount of intermediate data needs to be computed at each layer.

[0123] This leads to an important observation: overlapping data constitutes the non-disposable portion of the data. The remainder of each tile constitutes the disposable portion of the data. This means that overlapping data is needed not only to compute the output data of the ending layer of the current tile, but also subsequently to compute the output data of the ending layer of subsequent tiles. Conversely, the remaining data in each tile is necessary for computed to the output data of the ending layer of the current tile, but can be deleted / rewritten afterward.

[0124] To consider this in detail using a simplified one-dimensional example, refer to... Figure 8 . Figure 8 The diagram illustrates the individual data elements in the input and output data of each of the three layers (layer 0, layer 1, and the final layer) of a neural network. Each circle represents a data element (which could be a pixel in, for example, an image or feature map). Each element depends on elements from a subgroup in the previous layer. Figure 8The connecting lines in the diagram indicate the elements of the previous layers that a given element depends on. Therefore, for example, the 3rd element in the output of the ending layer depends on the 2nd, 3rd, and 4th elements in the output of layer 1. The 4th element in the output of the ending layer depends on the 3rd, 4th, and 5th elements in the output of layer 1. In other words, in this example, the kernel size of the ending layer is 3, and the stride is 1. The kernel size of layer 1 is 3, and the stride is 2. The kernel size of layer 0 is 3, and the stride is 1. Since the stride of both layer 0 and the ending layer is 1, the output of layer 0 is approximately the same size as the input data, and the output of the ending layer is approximately the same size as the output of layer 1 (depending on padding needs). Meanwhile, since the stride of layer 1 is 2, the output of layer 1 is approximately half the output of layer 0. The dashed lines represent the boundaries between blocks (tiles). That is, the 3rd data element shown in the ending layer is the final data element of the first block (tile 1). Taking the 4th element in the final layer output as an example, it can be seen that each element in the final layer output depends on: the 3 elements in the output of layer 1; the 7 elements in the output of layer 0; and the 9 elements in the input data. The elements in tile 1 (except for the overlapping element 1) are shown with diagonal shading. Three such elements (1-3) are shown in the final layer output as examples.

[0125] Due to data dependencies, some elements can be discarded after the final layer output of the current block (tile 1) has been computed. These elements are called "disposable" elements. Some other elements will also be used to compute the final layer output of the next block (tile 2). These elements are called "non-disposable" elements. Specifically, overlap 1 elements ( Figure 8 Elements 7 and 8 in the input data are non-one-time elements. Elements 6 and 7 in the output of layer 0 are used to calculate the final layer output data of the current block (block 1) during the first pass through multiple layers. However, these elements are also needed to calculate elements 8 and 9 in the output of layer 0. Elements 8 and 9 are not needed when calculating the first block during the first propagation, but they will be needed when calculating the second block during the second propagation. Therefore, to minimize memory access overhead, it is desirable to retain elements 7 and 8 of the input data in the on-chip memory 314 to avoid needing to read these elements from DDR 312 again during the second propagation. Elements 7 in the output of layer 0 and elements 3 and 4 in the output of layer 1 are also overlapping elements, because these elements will also be needed during the calculation in the second propagation (elements 7 in the output of layer 0 will be needed to calculate elements 5 in the output of layer 1; and elements 3 and 4 in the output of layer 1 will be needed to calculate elements 4 and 5 in the final layer output).

[0126] Now consider Figure 8The remaining elements of tile 1 are shown in the shading. Element 6 of the input data is used to compute elements 5, 6, and 7 of the layer 0 output. However, each of these three elements must be computed during the first propagation to compute the first output block (tile 1). Element 6 of the input data is not needed in any additional computations during the second propagation; therefore, this element is "one-time" data. Similarly, element 6 in the layer 0 output is used to compute element 4 in the layer 1 output, but this element is no longer needed during the second propagation (provided that element 4 in the layer 1 output is retained).

[0127] Elements of Tile 2 (excluding overlapping 2 elements) are shown as hollow circles. Overlapping 2 elements are shown in gray shading. These elements will be reused in the calculation of Tile 3, and are therefore non-disposable data. Some Tile 3 elements are shown at the bottom of the figure with horizontal shading.

[0128] The inventors of this invention have recognized that one-time data is only used to compute the next layer in the current propagation process, which involves multiple layers. Once the next layer in the current propagation process has been computed, one-time data from previous layers can be deleted / rewritten. For all layers in the current propagation process, non-one-time data (overlapping) must be retained so that it can be used for computation of the second block in the second propagation process. This recognition reduces unnecessary read / write operations to / from DDR 314 and also reduces unnecessary recomputation of intermediate output data.

[0129] Now refer to Figure 9 This describes a memory management strategy based on an example of a hardware implementation. The inventors have recognized that one-time data is relatively large but only valid for a short period of time. Conversely, non-one-time data (overlapping) is relatively small but valid for the entire propagation process. Figure 9 An exemplary memory allocation of on-chip memory 314 utilizing these characteristics is shown. In this example, it is assumed that six layers (numbered 0, 1, 2, 3, 4, and 5) of a neural network will be processed together. In one branch, each layer is connected to its preceding layer. In a second branch, layer 5 is connected to layer 1. This means that the output data of layer 1 is a portion of the input data of layer 5.

[0130] The memory management strategy divides the on-chip memory 314 into three segments. The first segment 802 (which may be referred to as "swap" memory) is used to store one-time data. The second segment 804 (which may be referred to as "heap" memory) is used to store non-one-time data. The third segment 806 (coefficient memory) is used to store weight data for each layer. Within the first segment 802, multiple slots A, B, and C are defined. During a given propagation process (for a given block of output data), the method is as follows: The one-time output of layer 0 is written to slot A. The non-one-time output of layer 0 is written to the beginning of the second segment 804. Data is read from these two locations before evaluating layer 1. During the evaluation of layer 1, the one-time output of layer 1 is written to slot B, and the non-one-time output of layer 1 is written to the second segment 804, for example, appended to the non-one-time output of layer 0. At this point, one-time data from layer 0 is no longer needed. Therefore, during the evaluation of layer 2, the contents of slot A can be rewritten using the one-time output of layer 2. The non-one-time output of layer 2 is appended to the data in the second section 804. However, neither slot B nor slot A should be overwritten when evaluating layer 3. Due to the branching, the contents of slot B (the one-time output of layer 1) are needed again to evaluate layer 5. Of course, the contents of slot A are immediately needed to evaluate the current layer (layer 3). Therefore, the one-time output of layer 3 is written to slot C. The one-time output of layer 4 is written to slot A, thus overwriting the contents of that slot that are no longer needed for further computation. To evaluate layer 5, the one-time output data of layer 1 is read from slot B, and the one-time output data from layer 4 is read from slot A. (Non-one-time output data from layers 1 and 4 is read from the second section 804.)

[0131] In this way, slots used for storing one-off data are frequently reused during a given propagation process, thus helping to reduce the total amount of on-chip memory 314 occupied at any given time. The size of each slot can be selected based on the maximum size of one-off data to be stored in the slot during the propagation process. Note that in this example, the second segment 804 is double-buffered. The non-one-off output data (overlap) of all layers of the current block is stored in one set of memory locations. During the next propagation process (for computing the next block), the overlap is retrieved from this set of memory locations as needed. The overlap for the next block is stored in a second set of memory locations. For subsequent propagation processes (subsequent blocks), the hardware implementation alternates between using the first and second sets of memory locations, i.e., reading from one set while writing to the other.

[0132] One-time and non-one-time portions of data can be identified by analyzing the structure of each of the multiple layers. Similarly, the allocation of one-time data from each layer to a specific slot in the first segment 802 of on-chip memory 314 can be accomplished by analyzing the overall layer structure of the multiple layers. This analysis can be performed dynamically by software at runtime or when mapping a given neural network to a given hardware implementation. Alternatively, this can be manually defined by the designer of the hardware implementation. However, the use of software facilitates mapping a wide variety of different neural networks to a given hardware implementation.

[0133] As mentioned above, the multiple layers of a neural network evaluated together in each propagation can include all layers of the neural network, but this is not necessary. Multiple layers can consist of a subset of the layers of the entire neural network. Depending on the structure of the neural network, advantageous and different layer groupings can be found. Figure 10 A simple neural network is shown, consisting of four layers with no branches, each layer depending solely on the output of the previous layer. This allows for a high degree of freedom in choosing how to group the layers together. For example, layers can be grouped into one or more groups according to any of the following grouping strategies:

[0134] ● 1-4

[0135] ● 1-2, then 3-4

[0136] ● 1-2, then 3, then 4

[0137] ● 1-3, then 4

[0138] ● 1, then 2-4

[0139] ● 1, then 2-3, then 4

[0140] ● 1, then 2, then 3-4

[0141] Generally speaking, given the size constraints of the on-chip memory 314, it may be preferable to group as many layers as possible together.

[0142] Figure 11A A slightly more complex network structure is shown. Again, there are four layers, each depending on the previous layer. However, there is an additional branch from layer 2 to layer 4. Figure 11AIn the diagram, the network is divided into two groups, each with two layers (layers 1 to 2 and layers 3 to 4), indicated by the dashed boxes. However, it should be noted that this may not be the optimal choice in terms of memory access requirements. The outputs of layers 1 to 2 will be written to DDR 312. When evaluating layer 3 (the first layer of layers 3 to 4), this output will be read from DDR 312; however, the evaluation of layer 4 also requires the output of layer 2. Therefore, the same data must be read from DDR 312 again to evaluate layer 4. Conversely, Figure 11B An alternative grouping of layers is shown, which may be more efficient in terms of memory access requirements. Here, layer 1 is processed by itself, and layers 2 through 4 are grouped into multiple layers. In this way, both branches are captured within multiple layers. The output data of layer 1 from DDR 312 is read once as the input data of the first layer (layer 2) of the multiple layers. Using the above reference... Figure 9 The memory management strategy described above can process the output data of layer 2 in multiple layers.

[0143] First, refer to the above Figure 2 Examples of convolutional layers in neural networks are given, in multiple p This operation is performed on two-dimensional (2D) image data in a plane or feature map. Exemplary hardware implementations according to the invention are indeed useful for such neural network layers. However, this disclosure is not limited to this approach. The above principles can be applied to 1D, 2D, or higher-dimensional data. When applied to 2D (or higher-dimensional) data, blocks of input and output data can be constructed in various ways. Each block can be defined in one, two, or more dimensions. For example, for 2D image data, blocks defined in one dimension can form 2D stripes (along a horizontal or vertical direction). Overlapping data can be defined above and below the stripes (for horizontal stripes) or to the left and right of the stripes (for vertical stripes). With this in mind, a plot is drawn... Figure 5 , Figure 6 and Figure 7 The diagrams. (These diagrams also apply to 1D blocks defined in 1D data.) It should be understood that, although... Figures 5 to 8 The instruction indicates that the data should be divided into blocks along the Y (height) dimension, but this is not limiting. As mentioned above, the same approach can be applied along the X (width) dimension. However, in some cases, it may be preferable to split along another dimension than to split along one dimension. For example, if the data is stored in memory row by row and the individual data elements are not neatly aligned with the burst size used for reading / writing to memory, then splitting the blocks along the Y (height) dimension may be preferred.

[0144] Alternatively, for 2D image data, blocks can be defined in two dimensions to form a 2D rectangle of data. Overlapping data can then be defined above, below, to the left, and to the right of this rectangle. The overlapping data from the current block may be useful for computing multiple subsequent blocks of output data, such as the block to the right of the current block, the block below the current block, and the block to the lower right of the current block (assuming the blocks are processed in raster scan order). The same principle can be extended to blocks in more dimensions that have overlapping data in multiple directions in multidimensional space.

[0145] The processing system may include the hardware implementation described above, or may be configured to perform the methods described above. The processing system may be an artificial intelligence accelerator system, such as a neural network accelerator (NNA), or a graphics processing system / graphics processing unit (GPU).

[0146] Figure 12 A computer system in which a processing system (e.g., an NNA) as described above can be implemented is shown. The computer system includes a CPU 902, an NNA 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 922. A processing block 910 (corresponding to hardware implementation 300) is implemented on the NNA 904. In other examples, the processing block 910 may be implemented on the CPU 902. Components of the computer system can communicate with each other via a communication bus 920. A memory bank 912 (corresponding to DDR 312) is implemented as part of the memory 906.

[0147] Figure 3 The hardware implementation 300 is shown as comprising a number of functional blocks. This is merely illustrative and not intended to define a strict division between different logical elements of such an entity. Each functional block may be provided in any suitable manner. It should be understood that intermediate values ​​described herein as being formed by the hardware implementation do not need to be physically generated by the hardware implementation at any point in time, and may only represent logical values ​​that conveniently describe the processing performed by the hardware implementation between its inputs and outputs.

[0148] The hardware implementations described herein can be embodied in hardware on an integrated circuit. The hardware implementations described herein can be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., a fixed logic circuit system), or any combination thereof. The terms “module,” “function,” “component,” “element,” “cell,” “block,” and “logic” are used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, cell, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein can be executed by one or more processors that execute code that causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other techniques to store instructions or other data and can be accessed by a machine.

[0149] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code for a processor, comprising code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code defining integrated circuits (e.g., hardware description languages ​​or netlists), and code expressed in programming languages ​​such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports the executable code to perform tasks specified by said code.

[0150] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or a collection or part thereof, that has the processing power to execute instructions. A processor can be any kind of general-purpose or special-purpose processor, such as a CPU, GPU, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0151] This invention also intends to cover software defining the configuration of hardware as described herein, such as hardware description language (HDL) software, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which computer-readable program code in the form of an integrated circuit definition dataset is encoded, configuring the system, when processed (i.e., run) in an integrated circuit manufacturing system, to manufacture a hardware implementation configured to perform any of the methods described herein, or a hardware implementation including any device described herein. The integrated circuit definition dataset may, for example, be an integrated circuit description.

[0152] Therefore, a method for manufacturing the hardware implementation as described herein can be provided at an integrated circuit manufacturing system. Furthermore, an integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, enables the method for manufacturing the hardware implementation to be executed.

[0153] Integrated circuit definition datasets can be in the form of computer code, such as netlists, code for configuring programmable chips, or hardware description languages ​​suitable for manufacturing at any level in integrated circuits, including register-transfer level (RTL) code, high-level circuit representations (such as Verilog or VHDL), and low-level circuit representations (such as OASIS (RTM) and GDSII). Higher-level representations (such as RTL) that logically define hardware suitable for manufacturing in integrated circuits can be processed on a computer system configured to generate manufacturing definitions of integrated circuits within the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate the manufacturing definition of the integrated circuit defined by that representation. As is typically the case where software executes at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate the manufacturing definition of the integrated circuit, executing code that defines the integrated circuit in order to generate the manufacturing definition of the integrated circuit.

[0154] Now refer to Figure 13 This describes an example of processing integrated circuit definition datasets at an integrated circuit manufacturing system in order to configure the system to manufacture hardware implementations or processing systems as described above.

[0155] Figure 13An example of an integrated circuit (IC) manufacturing system 1002 is shown, which is configured to manufacture hardware implementations or processing systems as described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining hardware implementations or processing systems as described in any of the examples herein), process the IC definition dataset, and generate ICs based on the IC definition dataset (e.g., embodying hardware implementations or processing systems as described in any of the examples herein). Through the processing of the IC definition dataset, the IC manufacturing system 1002 is configured to manufacture integrated circuits embodying hardware implementations or processing systems as described in any of the examples herein.

[0156] The layout processing system 1004 is configured to receive and process an IC definition dataset to determine a circuit layout. Methods for determining a circuit layout based on an IC definition dataset are known in the art and may involve, for example, synthesizing RTL code to determine the gate-level representation of the circuit to be generated, for example, in relation to logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user intervention to optimize the circuit layout. Once the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0157] As is known in the art, IC generation system 1006 generates ICs according to a circuit layout definition. For example, IC generation system 1006 can implement a semiconductor device manufacturing process for generating ICs, which may involve a multi-step sequence of photolithography and chemical processing steps, during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which can be used in the photolithography process to generate ICs according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code, which IC generation system 1006 can use to form a suitable mask for generating ICs.

[0158] The various processes performed by the IC manufacturing system 1002 can all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1002 can be a distributed system, allowing some processes to be performed in different locations and by different parties. For example, some of the following stages can be performed in different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition dataset to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0159] In other examples, processing of an IC definition dataset at an IC manufacturing system can configure the system to manufacture a hardware implementation or processing system without processing the IC definition dataset to determine circuit layout. For example, an IC definition dataset can define the configuration of a reconfigurable processor, such as an FPGA, and processing of said dataset can configure the IC manufacturing system (e.g., by loading configuration data into the FPGA) to generate a reconfigurable processor with said defined configuration.

[0160] In some implementations, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition dataset can enable the integrated circuit manufacturing system to generate devices as described herein. For example, the integrated circuit manufacturing definition dataset, as described above, relates to... Figure 13 The configuration of the integrated circuit manufacturing system described herein can produce devices as described in this document.

[0161] In some examples, an integrated circuit definition dataset may include software running on hardware defined at the dataset, or software running in combination with hardware defined at the dataset. Figure 13 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset during the manufacturing of the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0162] Compared to known implementations, the implementation of the concepts set forth in this application in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) can lead to performance improvements. Performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), trade-offs can be made between performance improvements and physical implementation methods, thereby improving manufacturing methods. For example, a trade-off can be made between performance improvements and layout area, thereby matching the performance of known implementations but using less silicon. This can be accomplished, for example, by reusing functional blocks serially or sharing functional blocks among elements of a device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that lead to improvements in the physical implementation of devices, apparatuses, modules, and systems (such as reduced silicon area) can be traded off for performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0163] The applicant has independently disclosed each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on the specification as a whole, in accordance with the common knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein. In view of the foregoing description, those skilled in the art will understand that various modifications can be made within the scope of this invention.

Claims

1. A method comprising multiple layers (102-1, 102-2, 102-3); A method for processing data in the hardware implementation of a neural network (100) of 502-1 and 502-2, the method comprising: Weight data representing the weights of the plurality of layers is provided in one or more first memory devices (312); Input data (104; 504) of the first layer (102-1; 502-1) of the plurality of layers is provided in the one or more first memory devices (312), the plurality of layers terminating at the generation p The final layer (102-3; 502-2) of each output dataset (106; 508) includes... n There are 1 output data element, of which n >2 and p ≥1, and p It depends on the weight data; A first subset of the input data (104; 504) representing a first block of the input data is read from the one or more first memory devices (312); Read the weight data from the one or more first memory devices; Process the weight data and the first subset of the input data to evaluate each of the plurality of layers, thereby computing a first block of output data for a first output dataset, wherein the first block of output data has m There are 1 output data element, of which m >1 and m < n ;as well as The first block of output data is written to the one or more first memory devices (312). At least one of the plurality of layers is a convolutional layer. The method further includes providing one or more second memory devices (314), wherein the cost of accessing the one or more second memory devices is less than the cost of accessing the one or more first memory devices. The first subset of processing the weight data and the input data includes: The first subset of the weight data and the input data (104; 504) is processed to evaluate the first layer of the plurality of layers, thereby calculating the output data (506) of the first layer. The output data (506) of the first layer is written into the one or more second memory devices (314); and For each subsequent layer among the plurality of layers: Read the output data of the previous layer from the one or more second memory devices (314); and The weight data and the output data of the previous layer are processed to evaluate the current layer among the plurality of layers, thereby calculating the output data of the current layer, and The method further includes, for at least one of the plurality of layers: When calculating the output data of the at least one layer: Identifying a one-time portion of the output data for the at least one layer, which should be deleted after calculating the first block of the output data for the final layer; and Identify the non-one-time portions of the output data of the at least one layer. These non-one-time portions should be retained after calculating the first block of the output data of the final layer for use in calculating at least one other block of the output data of the final layer. The method includes, when the output data of the at least one layer is written to the one or more second memory devices (314): Write the one-time portion into the first segment (802) of the one or more second memories; and The non-one-time portion is written into a second different segment (804) of the one or more second memory devices.

2. The method of claim 1, wherein when evaluating a subsequent layer among the plurality of layers, at least a portion of the one-time portion of the output data is rewritten, wherein the one-time portion is rewritten by the one-time portion of the output data of the subsequent layer.

3. A hardware implementation (300) of a neural network (100) comprising multiple layers (102-1, 102-2, 102-3; 502-1, 502-2), wherein at least one layer is a convolutional layer, the hardware implementation comprising: One or more first memory devices (312), the one or more first memory devices being configured to store: Weight data representing the weights of the multiple layers; and The input data (104; 504) of the first layer (102-1; 502-1) of the plurality of layers, the plurality of layers terminating at generation p The final layer of the output datasets (106; 508) (102-3; 502-2), each set includes n There are 1 output data element, of which n >2 and p ≥1, and p Depending on the weighted data, An input buffer (310) is configured to receive a first subset of the input data (104; 504) representing a first block of the input data. A coefficient buffer (308) is configured to obtain the weight data; One or more processing elements (318) are configured to process the weight data and the input data (104); The first subset of 504) is used to evaluate each of the plurality of layers, thereby computing a first block of the output data (106; 508) of the first output dataset, wherein the first block of the output data has m There are 1 output data element, of which m >1 and m < n ; as well as Output buffer (316), the output buffer being configured to output the first block of output data. The hardware implementation further includes one or more second memory devices (314), wherein the cost of accessing the one or more second memory devices is less than the cost of accessing the one or more first memory devices. The one or more processing elements (318) are configured to process the weight data and the first subset of the input data to evaluate the first layer of the plurality of layers, thereby calculating the output data of the first layer; The output buffer (316) is configured to write the output data of the first layer into the one or more second memory devices (314). as well as For each subsequent layer among the plurality of layers: The input buffer is configured to read the output data of the previous layer from the one or more second memory devices (314); and The one or more processing elements (318) are configured to process the weight data and the output data of the previous layer to evaluate the current layer among the plurality of layers, thereby calculating the output data of the current layer. Wherein, when the output data of at least one of the plurality of layers has been calculated, the output buffer (316) is configured as follows: The one-time portion of the output data of the at least one layer shall be deleted after the first block of the output data of the final layer is calculated; Identify the non-one-time portion of the output data of the at least one layer, and retain the non-one-time portion after calculating the first block of the output data of the final layer for use in calculating at least one other block of the output data of the final layer; Write the one-time portion into the first segment (802) of the one or more second memory devices (314); and The non-one-time portion is written into different second segments (804) of the one or more second memory devices (314).

4. The hardware implementation as described in claim 3, wherein, When evaluating a subsequent layer among the plurality of layers, the output buffer (316) is configured to rewrite at least a portion of the one-time portion of the output data, wherein the output buffer (316) is configured to rewrite the one-time portion with the one-time portion of the output data of the subsequent layer.

5. The hardware implementation as described in claim 3 or 4, wherein the second segment (804) of the one or more second memory devices (314) is double-buffered.

6. The hardware implementation as described in claim 3 or 4, wherein, When evaluating subsequent layers among the plurality of layers: The input buffer (310) is configured to read the one-time portion from the first segment (802) of the one or more second memory devices (314); The input buffer (310) is configured to read the non-one-time portion from the second segment (804) of the one or more second memory devices (314); and The one or more processing elements (318) are configured to use the one-time portion and the non-one-time portion in the calculation of the output of the subsequent layer.

7. The hardware implementation as described in claim 3 or 4, wherein: The input buffer (310) is configured to obtain a second subset of the input data representing a second block of the input data; The input buffer (310) is configured to read a non-one-time portion of the output data from at least one of the plurality of layers from the one or more second memory devices (314), the non-one-time portion having previously been written to the one or more second memory devices during the computation of the first block of the output data of the first output dataset; The one or more processing elements (318) are configured to process the weight data, the non-one-time portion, and the second subset of the input data to evaluate the plurality of layers, thereby computing a second block of the output data of the first output dataset; and The output buffer (316) is configured to output the second block of output data.

8. A processing system (904) configured to perform the method as claimed in claim 1 or 2, and / or include a hardware implementation (300, 910) as claimed in any one of claims 3 to 7.

9. A method for manufacturing a hardware implementation (300) as described in any one of claims 3 to 7 or a processing system (904) as described in claim 8 using an integrated circuit manufacturing system (1002), the method comprising: The computer-readable description of the hardware implementation or processing system is processed using a layout processing system to generate a circuit layout description of the integrated circuit embodying the hardware implementation or processing system; and The hardware implementation or processing system is manufactured using an integrated circuit manufacturing system based on the circuit layout description.

10. A computer-readable storage medium storing computer-readable code configured to perform the method of claim 1 or 2 when the code is executed.

11. A computer-readable storage medium storing a computer-readable description of a hardware implementation as described in any one of claims 3 to 7 or a processing system as described in claim 8, wherein the computer-readable description, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to perform the following operations: The computer-readable description of the hardware implementation or processing system is processed using a layout processing system to generate a circuit layout description of an integrated circuit embodying the hardware implementation or processing system; and The hardware implementation or processing system is manufactured using an integrated circuit manufacturing system based on the circuit layout description.

Images