Hardware and non-transitory computer readable storage media implementing convolutional neural networks

Abstract

This application relates to hardware and non-transitory computer readable storage media that implement convolutional neural networks. A method of hardware implementation of a convolutional neural network (CNN), the method comprising: receiving a first subset of data comprising at least a portion of weight data for a CNN layer and at least a portion of input data, and performing a convolution of the first subset of data using at least one convolution engine to generate a first partial result; receiving a second subset of data comprising at least a portion of the weight data for the CNN layer and at least a portion of the input data, and performing a convolution of the second subset of data using the at least one convolution engine to generate a second partial result; and combining the first partial result and the second partial result to generate at least a portion of convolution data for the CNN layer.

Description

[0001] This application is a divisional application of the invention patent application with application number 201710304906.6 (filing date: May 3, 2017, original invention name: Hardware implementation of convolutional neural network). TECHNICAL FIELD

[0002] The present application relates to hardware implementation of convolutional neural networks, and more particularly to methods and hardware for implementing convolutional neural networks, manufacturing methods and systems. BACKGROUND

[0003] Convolutional neural networks (CNNs) are a type of artificial neural network that can be used for machine learning applications. In particular, CNNs can be used in signal processing applications, including image processing and computer vision applications.

[0004] CNNs have been implemented in applications where power resources are not a significant factor. Nonetheless, CNNs have been applied in several different technical fields where resources of hardware used to implement the CNNs are limited in power consumption, processing capability, or silicon area.

[0005] Accordingly, there is a need for hardware configured to implement CNNs in an efficient manner, i.e., in a manner that requires less silicon area or less processing capability when operating. Moreover, the definition of a CNN for a particular application can change over time. For example, additional training can result in a changed definition of a CNN. Accordingly, there is also a need for hardware for implementing CNNs to accommodate changing requirements of CNNs. SUMMARY

[0006] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] A method in hardware implementation of a convolutional neural network (CNN) is provided, the method comprising: receiving a first subset of data comprising at least a portion of weight data for a layer of the CNN and at least a portion of input data, and performing a convolution of the first subset of data using at least one convolution engine to generate a first partial result; receiving a second subset of data comprising at least a portion of the weight data for the layer of the CNN and at least a portion of the input data, and performing a convolution of the second subset of data using the at least one convolution engine to generate a second partial result; and combining the first partial result and the second partial result to generate at least a portion of convolution data for the layer of the CNN.

[0008] The first subset of data can include a first portion of input data and weight data; and the second subset of data can include a second portion of input data and weight data. The second portion of input data can include a subset of the first portion of input data, and a size of the subset of the first portion of input data can be based on a size of the convolution kernel. The first subset of data and the second subset of data can each include all weight data for the particular layer.

[0009] The first subset of data can include a first portion of weight data and input data; and the second subset of data can include a second portion of weight data and input data. The first subset of data and the second subset of data can each include all input weight data for the particular layer. The first portion of weight data can include a different portion of weight data for the layer than the second portion of weight data. Combining the first portion result and the second portion result can include writing the first portion result and the second portion result to memory.

[0010] The first subset of data can include a first portion of weight data and input data; and the second subset of data can include a second portion of weight data and input data. Combining the first portion result and the second portion result to generate at least a portion of convolution data for a layer of the CNN can include: performing a convolution of the first portion of weights with the input data using the at least one convolution engine to generate the first portion result; performing a convolution of the second portion of weights with the input data using the at least one convolution engine to generate the second portion result; placing the first portion result in an accumulator buffer; combining the first portion result with the second portion result in the accumulator. The first portion of weights and the second portion of weights can be non-overlapping portions of the same weights.

[0011] The input data of the first subset of data and the input data of the second subset of data can include all input data for the layer.

[0012] A hardware implementation of a convolutional neural network (CNN) is provided, the hardware implementation comprising: an interface configured to receive a first subset of data comprising at least a portion of weight data and at least a portion of input data for a layer of the CNN, and configured to receive a second subset of data comprising at least a portion of weight data and at least a portion of input data for the layer of the CNN; at least one convolution engine configured to perform a convolution of the first subset of data to generate a first portion result, and configured to perform a convolution of the second subset of data to generate a second portion result; and wherein the hardware implementation is configured to combine the first portion result and the second portion result to generate at least a portion of convolution data for the layer of the CNN.

[0013] There is provided a configurable activation module for a CNN configured to implement one or more user-defined activation functions, the activation module configured to: receive configuration data specifying one or more characteristics of an activation function to be applied to input data; based on the received configuration data, store data representing the activation function in entries of a lookup table; receive input data representing an input to the activation function, and based on at least a portion of the received input, read one or more entries from the lookup table; and based on the one or more entries read from the lookup table, compute an output value representing an output of the activation function.

[0014] Hardware implementations of a CNN or activation module for a CNN can be implemented in hardware on an integrated circuit. There can be provided a method of manufacturing a hardware implementation of a CNN or activation module for a CNN in an integrated circuit manufacturing system. There can be provided a set of integrated circuit definition data which, when processed in an integrated circuit manufacturing system, configures the system to manufacture a hardware implementation of a CNN or activation module for a CNN. There can be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an integrated circuit which, when processed, causes a layout processing system to generate a circuit layout description for use in an integrated circuit manufacturing system to manufacture a hardware implementation of a CNN or activation module for a CNN.

[0015] There can be provided an integrated circuit manufacturing system comprising: a non-transitory computer readable storage medium having stored thereon a computer readable integrated circuit description describing a hardware implementation of a CNN or activation module for a CNN; a layout processing system configured to process the integrated circuit description in order to generate a circuit layout description of an integrated circuit implementing a hardware implementation of a CNN or activation module for a CNN; and an integrated circuit generation system configured to manufacture a hardware implementation of a CNN or activation for a CNN from the circuit layout description.

[0016] There can be provided computer program code for performing a method according to any preceding claim. There can be provided a non-transitory computer readable storage medium having stored thereon computer readable instructions which, when executed at a computer system, cause the computer system to perform a method according to any preceding claim.

[0017] As will be apparent to the skilled person, the above features can be combined as appropriate and with any of the aspects of the examples described herein. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 An example of the structure of data used in a convolutional neural network is shown;

[0019] Figure 2 An example hardware implementation of a convolutional neural network is shown;

[0020] Figure 3 An example of a convolution engine is shown; Figure 2

[0021] Figure 4a An example method for operating an activation module is shown;

[0022] Figure 4b A rectified linear unit (ReLU) function that can be implemented in the activation module shown in any of Figure 2 , Figure 4d and Figure 4f is shown;

[0023] Figure 4c A parametric rectified linear unit (PReLU) function that can be implemented in the activation module shown in any of Figure 2 , Figure 4d and Figure 4f is shown;

[0024] Figure 4d An example hardware implementation of the activation module shown in Figure 2 is shown;

[0025] Figure 4e An example of interpolation performed in the activation module of Figure 4f is shown;

[0026] Figure 4f Another example hardware implementation of the activation module shown in Figure 2 is shown;

[0027] Figure 4g Another example hardware implementation of the activation module shown in Figure 2 is shown;

[0028] Figure 5 An example method of processing data in a hardware implementation of a CNN of Figure 2 is shown;

[0029] Figure 6 An example method of processing data in a hardware implementation of a CNN of Figure 2 is shown;

[0030] Figure 7a An example division of an input data example for a layer into two or more input data subsets for processing is shown;

[0031] Figure 7b An example method of processing data of Figure 7a is shown;

[0032] Figure 8a ​An example method for processing data is shown in a hardware implementation of a CNN or activation module for a CNN is shown;

[0033] Figure 8b An example method for processing data is shown in a hardware implementation of a CNN or activation module for a CNN is shown; Figure 8a

[0034] Figure 9 An example method for processing data is shown in a hardware implementation of a CNN or activation module for a CNN is shown; Figure 2

[0035] Figure 10 An example method for processing data is shown in a hardware implementation of a CNN or activation module for a CNN is shown; Figure 2

[0036] Figure 11 An example method for processing data is shown in a hardware implementation of a CNN or activation module for a CNN is shown; Figure 2

[0037] Figure 12 An example method for processing data is shown in a hardware implementation of a CNN or activation module for a CNN is shown; Figure 2

[0038] Figure 13 A computer system in which a hardware implementation of a CNN or activation module for a CNN is implemented is shown; and

[0039] Figure 14 An integrated circuit manufacturing system for generating an integrated circuit that implements a CNN or hardware module for a CNN is shown.

[0040] The accompanying drawings illustrate various examples. The skilled person will understand that the illustrated element boundaries (e.g. boxes, groups of boxes, or other shapes) in the drawings represent one example of boundaries. One element can be designed as multiple elements or multiple elements can be designed as one element in some examples. Where appropriate, common reference numerals have been used throughout the drawings to refer to similar features. DETAILED DESCRIPTION

[0041] The following description presents examples by way of example to enable a person skilled in the art to make and use the application. The application is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0042] Embodiments will now be described by way of example only.

[0043] ​​​​​A Convolutional Neural Network (CNN) is a form of artificial neural network that consists of multiple interconnected layers with weights associated with them, enabling CNNs to perform signal processing tasks, including computer vision tasks. The functionality of a CNN is entirely represented by its weights. In some examples, a CNN can be configured to receive an input image.

[0044] exist Figure 1 The diagram illustrates an overview of examples of data formats used in CNNs. (For example, in...) Figure 1 As can be seen, the data format used in CNNs can be formed by multiple planes. The input data can be configured as P planes of data, where each plane has dimensions x×y. A CNN consists of multiple layers, each with multiple filters w0…w0 associated with it. n Filter w0…w n Each has dimensions m×n×P, and convolution operations are applied to the input data based on several steps across directions s and t, such as in Figure 1 As shown in the diagram.

[0045] As mentioned above, each layer can have multiple filters w0…w associated with it. n As used in this article, weights can also be referred to as filters, filter weights, or coefficients. The number and value of filter weights can vary between layers, such that for the first layer, the number of weights can be defined as... Furthermore, for the second layer, the number of weights can be defined as... The number of weights in the first layer is n1 and the number of weights in the second layer is n2.

[0046] For multiple layers of a CNN, the input data for that layer is processed by convolutional weights associated with that layer. For the first layer, "input data" can be considered the initial input to the CNN, which in some examples might be an image—for instance, where the CNN will be used for a vision application. The first layer processes the input data and generates a first intermediate dataset that is passed to the second layer. This first intermediate dataset can also take the form of several data planes. This first intermediate dataset can be considered as forming the input data for the second layer, which processes the first intermediate data to produce output data in the form of second intermediate data. In the case where the CNN includes a third layer, the third layer receives the second intermediate data as input data and processes that data to produce third intermediate data as output data. Therefore, a reference to input data can be interpreted to include a reference to the input data for any layer. For example, the term "input data" can refer to intermediate data, which is the output of a particular layer and the input to subsequent layers. This is repeated until the final layer produces output data that can be considered the output of the CNN.

[0047] Figure 2 A hardware implementation 200 configured to implement a CNN is illustrated in accordance with the present disclosure. The hardware implementation comprises digital logic circuitry configured to receive data defining a CNN to be implemented, and configured to define operations in accordance with that data to process input data in order to implement the CNN. Thus, the hardware implementation can be considered to be configurable hardware for implementing one or more CNNs. As such, references herein to hardware implementations of CNNs are references to hardware configured to implement a CNN by operating on input data in accordance with received configuration data. The hardware implementation itself is not necessarily configured to implement a particular CNN, and is not necessarily pre-loaded with specific configuration data for a CNN. For example, the CNN need not comprise pre-loaded (or hard-coded) weight data and layer definitions. Thus, the hardware implementation is configured to implement any CNN based on configuration data it receives. As such, the configuration data (also referred to herein as command data) defines the particular CNN to be implemented, including the number of layers to be included, the size and values of the weight data, and the desired format of the input data.

[0048] The data defining the CNN and how the CNN is to be processed can comprise configuration (or command) data, weight data, and input data. Thus, the hardware implementation is configured to process the input data using the weight data in order to perform the operations defined by the CNN. The hardware implementation 200 comprises a memory interface 210, an input buffer controller 215, a command decoder 220, a coefficient buffer controller 225, a coefficient buffer 230, n convolution engines 240, n accumulators 245, an activation module 255, a normalization module 265, a shared buffer 270, and a pooling module 275.

[0049] The memory interface 210 is configured to provide an interface between the hardware implementation 200 and external memory (not shown). The external memory can be considered to be a module separate from the hardware implementation 200 or can be considered to be integrated with the hardware 200. The command or configuration data can for example comprise information about the size and / or format of the weight data, the input data size and format, and their location in the external memory.

[0050] The memory interface 210 is configured to receive from the external memory the weights and data to be used in the calculations within the CNN, and command information to control the operation of the hardware implementation 200. The received weights (also referred to herein as coefficients) are passed to the coefficient buffer controller 225, and the received data is passed to the input buffer controller 215. The received commands are passed to the command decoder 220, which in turn is configured to decode the commands, and subsequently issue control information to the elements of the hardware implementation (including the coefficient buffer controller 225 and the input buffer controller 215) to control the manner in which the weight and input data are stored in the buffers.

[0051] The weights and input data received from the external memory via the memory interface 210 during a read of the external memory can form weights and input data for only a portion of a single layer, all of the weights and input data to be used in processing a single layer, or can include weights and input data for processing multiple layers. For example, the weights received from the external memory can form the weights of a single layer, and the input data received can form only a portion of the input data for a single layer (or vice versa). Any combination of data and weights across one or more layers can be received from the external memory in a single read from the memory (e.g., using a burst read).

[0052] In practice, the number of weights and data received in a single read from the external memory will depend on the size of the coefficient buffer 230 and the input buffer 235. The weights are passed from the coefficient buffer controller 225 to the coefficient buffer 230, and the received data is passed from the input buffer controller 215 to the plurality of input buffers 235a-235n. The number of input buffers will depend on the particular implementation of the hardware 200 but can take any value. The input data is shared across all of the input buffers 235a-235n. The input buffers each form an active bank, such that the number of input buffers can be increased or decreased depending on the application.

[0053] The input buffers 235a-235n are connected to each of a plurality of multiplexers, as each convolution engine 240a-240n requires access to all of the active "banks" of input data. The multiplexers are each configured to select an output from one of the input buffers, and pass the value output from the selected input buffer to the corresponding convolution engine 240a-240n. In addition, the weights from the coefficient buffer 230 are provided as a second input into each convolution engine 240a-240n. The convolution engines are configured to perform a convolution operation on the received input data using the weights received from the coefficient buffer 230. The result output of each convolution engine 240a-240n is provided as an input to a corresponding accumulator of the plurality of accumulators 245a-245n. In Figure 3 An example implementation of a convolution engine 240n is illustrated in FIG. 3, which includes a plurality of elements of multiplication logic 310 and a plurality of elements of addition logic 320.

[0054] Each accumulator 245a-245n is connected to an accumulation buffer 250. The accumulation buffer 250 is configured to store the accumulated results received from each accumulator 245a-245n. The accumulation buffer 250 is connected to the memory interface 210. As such, the accumulation buffer 250 is configured to send data to and receive data from the external memory via the memory interface 210. In particular, the accumulation buffer 250 is configured to be able to store and restore its value from the external memory via the memory interface 210, as will be described in more detail below. The accumulation buffer 250 is connected to the inputs of the accumulators 245a-245n and is configured to feed back values to the accumulators 245a-245n to cause the accumulation computation to take place.

[0055] Activation module

[0056] The accumulation buffer 250 is configured to pass the accumulated values to an activation module 255. The activation module 255 is configured to perform at least one of a number of different activation functions. Activation modules configured to perform the methods illustrated in Figure 4a The activation module is an activation module for use in a CNN. The activation module can be configured to implement one or more user-defined activation functions. The activation module can be configured to receive, at step 401, configuration data specifying one or more characteristics of an activation function to be applied to input data. The configuration data can be used to specify, for example, one or more data points of the activation function, one or more features of the activation function, one or more values that can be used to derive or identify a particular activation function to be implemented in the activation module 255 for one or more passes of the activation function.

[0057] The activation module 255 can also be configured to store, at step 402, data representing the activation function in entries of a lookup table based on the received configuration data. In some examples, the data representing the activation function can be the received configuration data. In some arrangements, where the configuration data comprises data points of the activation function, the data points can themselves be stored in the lookup table. In some arrangements, the configuration data can be used to identify a pre-stored activation function, such that data representing that identified activation function, such as data representing data points of that pre-stored activation function, can be loaded into the lookup table - for example using an identifier in the configuration data. In some arrangements, the data stored in the lookup table can be gradient and offset values for at least part of the activation function. Thus, the values in the lookup table can represent an estimate or approximation of the activation function.

[0058] The activation module 255 can be configured to receive, at step 403, input data representing an input of an activation function, and can be configured to read one or more entries from a lookup table based on at least a portion of the received input. In some arrangements, the activation module 255 is configured to use at least a portion of the received input as an address into the lookup table. For example, a predefined number of values can be stored in the lookup table, and a value of the received input can be the value used as the address into the lookup table. For example, the address to be used can be determined from a predefined number of MSBs of the input data.

[0059] The activation module can be configured to compute, at step 404, an output value representing an output of the activation function based on the one or more entries read from the lookup table. The activation module can further compute the output value based on at least a portion of the input data (e.g. a predefined number of LSBs of the input data). The portion of the input data used to look up the one or more entries in the lookup table can be different from the portion of the input data used to compute the output value.

[0060] The activation module can compute the output value by interpolating between two or more entries read from the lookup table. For example, in the case that at least two entries are read from the lookup table, a linear interpolation can be performed using the two entries and at least a portion of the input data. For example, in the case that at least three entries are read from the lookup table, a quadratic interpolation can be performed using the three entries and at least a portion of the input data.

[0061] In some arrangements, the data points, the shape of the function or an identifier of the function to be implemented can be defined in the configuration data. For example, one of a number of predefined functions to be implemented can be identified.

[0062] In some implementations, the activation module 255 can be configured to operate as a rectified linear unit (ReLU) by implementing a ReLU function (which can be implemented by receiving control data defining the points of the activation function). In a ReLU function, the output element y i,j,k As set out below and as illustrated in Figure 4b

[0063] y i,j,k = f(x i,j,k ) = max{0, x i,j,k}

[0064] Thus, for x values less than 0, y = 0.

[0065] In some implementations, the activation module 255 can be configured to operate as a parametric rectified linear unit (PReLU) by implementing a PReLU function. The PReLU function performs a similar operation to the ReLU function. In particular, in a PReLU function, the output element y​ PReLU is configured to generate an output element y i,j,k As set forth below and as illustrated in Figure 4c

[0066] y i,j,k = f(x i,j,k ; w1, w2, b1, b2) = max{(w1*x i,j,k + b1), (w2*x i,j,k + b2)}

[0067] In some implementations, the activation module 255 can be configured to perform one or more other functions depending on the particular application. In some implementations, the activation module 255 can be configured to be programmed with the particular function to be performed. For example, the activation module 255 can be configured such that for a particular layer of a CNN, it receives data defining the function to be performed. The function to be implemented in the activation module (i.e., the activation function) can be considered a transfer function.

[0068] In one approach to representing an activation function, the entries in the lookup table can be formed from at least one gradient and at least one offset. For example, the received data can represent pairs of gradients and offsets. In this way, the activation module 255 can thus implement the activation function as a value and slope lookup table with linear interpolation used for the intermediate values. The input value (e.g., from the accumulation buffer) can be used as an address into the lookup table. In one example, where the lookup table is a 64 entry table, the 6 MSBs of the input can be used as the address into the lookup table. The lookup table can contain values for the sample C and gradient m at the point defined by the MSBs. In this case, the remaining bits of the input can be interpreted as a value between 0 and 1, such that the output of the activation module is defined by:

[0069] y = m * (LSBs) + C

[0070] An example hardware implementation of the activation module 255 of this arrangement is illustrated in Figure 4d . As illustrated in Figure 4d , the activation module 255 can include a lookup table 410 configured to store values representing the activation function to be implemented within the activation module 255. The activation module 255 also includes multiplication logic and addition logic to perform the calculations defined above. In particular, the MSBs of the input value are used as an address into the lookup table, and the corresponding m and C values are read from the lookup table and output. The remaining LSBs of the input data are then multiplied with the lookup table output m, and the result is then added to the lookup table C.

[0071] ​In another arrangement, instead of storing m and C values in a lookup table, which are then retrieved, two different coarse output values are determined that are positioned on either side of the input value. Based on at least a portion of the output value, the output value to be output from the activation module is determined by interpolating between the two coarse output values, as described below with reference to Figure 4e and Figure 4f In particular, this arrangement can involve storing a set of discrete output values in a lookup table. Using the 64 entry example described above, each entry can represent one of 64 different output values of the activation function. However, when an input value is received, the six MSBs of the input value can be used as an address in the table to look up a corresponding output value. In addition, the output value at the next address location in the lookup table can also be retrieved. As illustrated in Figure 4f , the MSBs of the input value can coincide with the input value x0, and it can be determined that the next corresponding input value is xi. Thus, the two values retrieved can be the corresponding output values for the identified input values (i.e. y0and yi). From the retrieved output values y0and yi, the LSBs are used to interpolate the input value x to determine an estimate for the corresponding output value y.

[0072] Figure 4f An example hardware arrangement configured to perform the interpolation method described above with reference to Figure 4e is illustrated. In this arrangement, the activation module 255 comprises a lookup table 410 configured to store output values that can be retrieved based on a lookup using a corresponding input value. The lookup table 410 is configured to receive data to be stored in the lookup table 410. In this way, the particular activation function to be performed by the activation module 255 is configurable. The activation module 255 also comprises an interpolation module 420 configured to perform an interpolation operation to calculate y. This interpolation can be implemented using the following equation:

[0073]

[0074] As illustrated in Figure 4f , the activation module 255 is configured to receive an input value x. A predefined number of n MSBs (which correspond to 2 n entries in the lookup table) are used as an address into the lookup table to retrieve an output value y0at a corresponding address in the lookup table and the next output value yi in the lookup table. These two values y0and yi are output from the lookup table 410 and input into the interpolation module 420, which performs an interpolation based on the remaining LSBs of the input value x to determine y. The interpolation module 420 then outputs the output value y that coincides with the input value x.

[0075] Another hardware implementation of the activation module 255 is illustrated in Figure 4g . In this implementation, the activation module 255 comprises a lookup table 410 and an interpolation module 420. The lookup table 410 is configured to store output values that can be retrieved based on a lookup using a corresponding input value. The lookup table 410 is configured to receive data to be stored in the lookup table 410. In this way, the particular activation function to be performed by the activation module 255 is configurable. The activation module 255 also comprises an interpolation module 420 configured to perform an interpolation operation to calculate y. This interpolation can be implemented using the following equation: Figure 4gIn this arrangement, the look-up table is configured to retrieve three output values (i.e. y0, yi and y2) based on the MSB of the input value x. This enables, for example, quadratic interpolation to be performed by the interpolation module 420. In this way, more complex curves can be interpolated by retrieving more coefficients from the look-up table 410. It will be appreciated that more than three values can be retrieved and used to interpolate the output value for the input value.

[0076] As mentioned above, the activation module 255 can be configured to receive a set of values defining an activation function to be performed. The received values can form the values to be retrieved from the look-up table 410, such that they can be loaded into the look-up table 410. In some implementations, the activation module 255 can be configured to receive the values defining the function to be performed from the command decoder 220. The command decoder 220 can be arranged to receive the values of the function from an external memory via the memory interface 210. In some implementations, the command decoder 220 can be configured to receive values defining the function to be used across multiple layers of the CNN implementation. In some implementations, the command decoder 220 can be configured to receive values defining the function to be used for only a single layer. Thus, for each layer to be processed, the command decoder 220 can receive values to be inserted into the look-up table 410 from an external memory. The command decoder 220 can then proceed to load a different set of values defining the function to be performed by the activation module 255 during processing of that layer into the look-up table 410 for each layer. In some arrangements, the activation module 255 can be configured to receive a different set of values to be stored in the activation layer for each hardware pass from the command decoder. In other words, a different activation function can be used to process each different input value. The command decoder can receive several different commands, and the command decoder can be configured to control the hardware implementation 200 such that the data being received by the activation module 255 conforms to the particular activation function being loaded into the activation module 255 at the time the data is received.

[0077] It will be appreciated that there are other methods of implementing activation functions based on the storage of values in a look-up table, which can be considered as coefficients of the activation function.

[0078] Standardization module

[0079] The result values calculated by the activation module 255 are passed to be processed by the normalisation module 265 via the shared buffer 270. The normalisation module 265 is configured to perform one of several different normalisation functions as set out below.

[0080] A Local Response Normalisation Layer function can be performed within the same plane of input data according to the following:

[0081]

[0082] In the above equation, is the activity of a cell in the map f at position x, y before normalization, S is the size of the input data (e.g., image data), k is a constant, and N is the size of the region used for normalization. The output dimension of this function is equal to the input dimension. In some examples, this module can be configurable in a similar manner as described above with respect to activation module 255. In particular, the normalization module can be configured to receive parameters a and β from command decoder 220 that have received such parameters from an external memory. In this way, the implementation of the local response normalization layer function can be modified for each layer or can be modified over the course of multiple layers.

[0083] The local response normalization layer function can also be performed across planes. The function is similar to the normalization function described above with some minor differences. In particular, a cell is divided by the activity of other cells in the same position, but in different planes. The parameter N indicates the number of neighboring filters used for normalization. For example, if a layer has 32 maps, then when N is set to 5, a cell in the 7th map can be normalized by cells in the 5th through 9th maps. The function used for the local response normalization layer function across planes is set forth below:

[0084]

[0085] In the above equation, F is the number of maps. As with the local response normalization layer between planes, the parameters a and β can be received from an external memory via command decoder 220.

[0086] The normalization functions described above can be implemented in hardware of normalization module 265 using the following equation:

[0087]

[0088] In the above equation, EXP is a hardware implementation of the exponential function e x Ln is a hardware implementation of the natural logarithm function ln(x), and C, k, and γ are constants received where C = -β and One possible implementation of the functions described above is a proof lookup table to perform the EXP and Ln functions in a similar manner as described above with respect to activation module 255. As previously described, the constants C, k, and γ can be received from an external memory via command decoder 220 and can be defined per layer or defined for a subset of layers. The normalization window size can be programmable per layer.

[0089] The normalization module 265 can also be configured to implement a local response normalization layer function defined as:

[0090]

[0091] In the above equation, is the average of all in the 2D field defined by the summation range. The local contrast normalization can be implemented, for example, using a modification of the filter weights or using an adjusted parametric ReLU operation.

[0092] The normalization module 265 can be configured to implement one or more of the normalization functions described above. In particular, the normalization module 265 can be configurable such that it is able to perform any of the three functions above and any other suitable function. In this way, the normalization module 265 can receive signaling to indicate which function the normalization module is to perform. This can be performed by using one of several pre-defined functions such that the control signaling received by the normalization function indicates which of the pre-defined functions the normalization module 265 should perform.

[0093] In other arrangements, the normalization module 265 is configured in a similar manner to the activation module 255 such that the normalization module 265 is able to receive values defining the values to be inserted into the lookup table. In this way, the normalization module 265 can be configured via external control. The control signaling can be received from an external memory via the command decoder 220.

[0094] It is not necessary for control signaling of the activation and normalization modules to be received via the command decoder 220 and it will be appreciated that the activation module 255 and the normalization module 265 can be controlled via another external interface or via a direct connection to the memory interface 210. Other arrangements are also possible.

[0095] Pooling

[0096] The results of the normalization function performed in the normalization module 265 are passed to the pooling module 275 via the shared buffer 270. In the pooling module 275, a pooling operation is performed. In some arrangements, the pooling operation occurs over a sliding window defined per layer. The operations performed in the pooling layer are similar to those performed in the convolution engine 240a-n. Examples of pooling functions include maximum value and average value calculations.

[0097] The results stored in the shared buffer 270 are passed to the memory interface 210 which can store the results in an external memory or pass the results back into the input buffer for further processing without having to be first passed out to the external memory.

[0098] The shared buffer 270 is configured for two purposes: to buffer values ​​from the activation module 255 until a correct sample is available for normalization and / or to buffer values ​​from normalization until a correct sample is available for pooling. In this way, the shared buffer is used for storage efficiency, since in many implementations normalization and pooling are not combined in the same process.

[0099] Normalization can be either a “same mapping” where data from NxN regions of the same plane is normalized, or a “cross-plane” where data is normalized within a window in the P-dimensional space. In the latter case, the filter always takes a single sample from each plane. When deciding how to subdivide larger layers for processing, the normalization pattern must be considered so that the shared buffer holds the correct data. If the size of the NxN window becomes too large for the parts to be held in the shared buffer, then normalization becomes a 2-pass operation. Statistics for normalization are focused on the write-out, and normalization is applied to the read-out of data for the following layers.

[0100] Configurability

[0101] Although Figure 2 The hardware implementation diagram illustrates the specific order in which the modules are arranged and how the data processing flows through the hardware, but it will be understood that the specific computations required and the order in which the data is processed across layers can vary.

[0102] In some layers, the functions executed by activation module 255, normalization module 265, and pooling module 275 can be used to perform activation, normalization, and pooling operations. In other layers, only one of these functions can be executed and not necessarily in the order described in hardware implementation 200. To achieve a configurable order of processing these functions, each of the activation module 255, normalization module 265, and pooling module 275 can be configured to receive control signaling to configure the module into a bypass mode, in which functions are not executed and input values ​​simply pass through the module without modification.

[0103] In some layers, data for a particular layer may need to be processed first by convolutional engines 240a-n and then second by activation module 255, normalization module 265, and pooling module 275. In these layers, the output from convolutional engines 240a-n is passed to accumulator buffer 250 via accumulators 245a-n and then to activation module 255, normalization module 265, and pooling module 275 for further processing. In other layers, data may need to be processed differently. For example, data may need to be processed first by activation module 255, normalization module 265, and pooling module 275 and then second by convolutional engines 240a-n.

[0104] In these arrangements, data can be passed directly to the activation module via the accumulation buffer 250, which has received input data directly from the memory interface 210 that has received data from the external memory. In this way, the processing performed by the convolution engines 240a-n and accumulators 245a-n is effectively skipped, and data can be passed directly to the activation module 255, normalization module 265, and pooling module 275. Then, once processing using the activation module 255, normalization module 265, and pooling module 275 is complete, the resulting values can be passed into the input buffer controller 215 via the memory interface 210. In some arrangements, the resulting values can first be passed to the external memory via the memory interface 210, and then retrieved from the external memory prior to use.

[0105] In other arrangements, the memory interface 210 can pass the resulting values to the input buffer controller 215 without passing the values to the external memory. By avoiding the need to pass values resulting from the calculations using the activation module 255, normalization module 265, and pooling module 275 to the external memory, memory bandwidth is reduced and thus latency in processing data is also reduced.

[0106] By providing the features described above, it is possible for the hardware implementation 200 to process various different configured CNNs by providing the ability to perform memory-to-memory computations to be performed. These memory-to-memory computations need not be directly related to a particular layer. In other words, the hardware implementation 200 is adaptable to the particular CNN being implemented. Advantageously, since the activation module 255, normalization module 265, and pooling module 275 are placed linearly, it is possible to perform these functions back-to-back without having to retrieve data from the external memory.

[0107] In some implementations, the order in which the activation module 255, normalization module 265, and pooling module 275 are connected can vary. For example, the activation module 255, normalization module 265, and pooling module 275 can be connected in reverse order, such that the pooling module is connected to the accumulation buffer 250, and the activation module is connected to the memory interface 210.

[0108] Partial processing

[0109] In typical hardware implementations of CNNs, it is desirable to reduce the number of accesses to required external memory while maintaining a reduced silicon footprint and also reducing the amount of power consumed in processing the CNN. However, in typical CNN applications, the amount of data to be processed varies between layers. Thus, CNN implementations often require a very large hardware implementation footprint to process each layer of a CNN to be processed, as the hardware implementation can be required to have buffers and processing capabilities to process the most processor-intensive layers, even if previous or subsequent layers require less processing or data storage. In other words, the buffers and computation logic should be large enough to be able to process any particular layer, including the layer with the largest weight size, the largest number of weights, and the largest amount of data. However, where a CNN is implemented in this way, the processing of smaller layers is inefficient, as the buffers and computation logic can not be fully utilized in other, smaller layers.

[0110] The hardware implementations of CNNs described herein provide flexible implementations that allow the CNN inputs (e.g., data and weights for each layer) to be split or segmented into several different parts that can be processed separately to produce partial results. These partial results can then be combined to produce the output from a particular layer. By segmenting or partitioning the data in the manner described below, a CNN can be implemented in a way that meets a silicon budget, at the expense of the ability to process different sizes of data, different numbers of weights, and sizes of weights for each layer.

[0111] Furthermore, it is preferable to minimize the number of memory accesses and writes performed when operating a CNN. The number of memory accesses and writes performed in the present hardware implementations can depend on the size of the data buffers and coefficient buffers. In particular, there are three scenarios in which the input for a particular layer of a CNN cannot be processed in a single read from memory. This can occur due to the input to a layer being incorrectly sized for the buffers implemented in hardware, and thus special processing of the input data needs to be provided.

[0112] Figure 5 A method 500 of processing data for a layer according to the present disclosure is illustrated. The method begins at step 510, where a first subset of data for a layer is received. The first subset of data can be defined by control data. At step 520, the first subset of data for the layer is processed to generate a first partial result. At step 530, a second subset of data for the same layer is received. At step 540, the second subset of data for the layer is processed to generate a second partial result. At step 550, the first partial result and the second partial result are combined.

[0113] In an example, the first subset of data and the second subset of data can be received from a memory. Figure 2The method 500 is implemented in the illustrated hardware implementation Figure 5 The first subset of data can be a subset of input data for a layer, a subset of weights for a layer, or a portion of at least one weight for a layer. In the case where the first subset of data is a subset of input data for a layer, the first subset is received in the input buffer 235a-235n at step 510 via the input buffer controller 215 and the memory interface 210. In the case where the first subset is a subset of weight data, the first subset is received at the coefficient buffer 230 via the coefficient buffer controller 225 and the memory interface 201.

[0114] At step 520, the first subset of data (i.e., the subset of input data, the subset of weights for a layer, or the portion of at least one weight for a layer) is processed to generate a first partial result. The processing involves passing the first subset of data and the corresponding weights or input data to one or more convolution engines 250a-250n (depending on the first subset of data). The convolution engines are configured to perform convolution operations between the weight data and the input data. Since the hardware implementation has a predefined number of convolution engines, the size of the first subset of data is preconfigured such that it is possible for the convolution engines to perform all convolution operations for the first subset of data in a single pass or in a predefined number of passes. For example, in the case where the first subset of data is a subset of input data, the input data can be split into batches, each of which can be processed in a single pass of the convolution engines. Other limitations on the first subset of data are the size of the coefficient buffer 230 and the size of the input buffers 235a-235n, and the identification of the first subset of data can also depend on the size of one or more of the input buffers 235a-235n and the coefficient buffer 230.

[0115] After the first subset of data has been processed at step 520, a partial result is generated. The generated partial result will depend on the type of data that was selected as the first subset of data. For example, the first subset of data can be a subset of input data. In this case, the first partial result can be the result of processing the first subset of input data with the weight data. Similarly, the first partial result can be the result of processing all of the input data with the subset of weight data (or the portion of at least one weight). This will be explained in more detail later. The first partial result can be held in the accumulators 245a-245n before being passed to the accumulation buffer 250.

[0116] At step 530, a second subset of data for the same layer is received in the same manner as the first subset of data. For example, where the first subset of data relates to weight data, the first subset of data is stored in the coefficient buffer. Where the second subset of data relates to input data, the second subset of data is stored in the input buffer 235a-235n.

[0117] At step 540, the second subset of data for the layer is processed to generate a second partial result. As with the first subset of data, the second subset of data is processed in the convolution engine 240a-240n. It will therefore be appreciated that the second subset of data is processed after the first subset of data using the same convolution engine. In this way, the first subset of data and the second subset of data are processed in separate passes of the convolution engine. At step 550, the first partial result and the second partial result are combined in an accumulator to form a combined result which is passed to the accumulation buffer 250 to form at least part of the output for the particular layer of the CNN. In this way, the processing of data for a layer is split into smaller batches of work which can be processed by hardware implementations which are too small to process all of the processing of a layer in a single pass.

[0118] Number of filter weights

[0119] As mentioned above, the first subset of data can be formed from a subset of weights.

[0120] It can be the case, for example, that the coefficient buffer 230 has a predefined size in the hardware implementation 200. The coefficient buffer 230 is arranged to store filter weights received via the coefficient buffer controller 225. For example, the coefficient buffer 230 can be required to provide 16x8 weights per clock to the convolution engines 240a-240n. In some arrangements, the coefficient buffer 230 can be sized such that it is able to store all m*n*p*f weights for a particular layer, where m and n define the size of the convolution kernel and f defines the number of weights for the particular layer. However, in other arrangements, the coefficient buffer can not be large enough to store all m*n*p*f weight values for a particular layer. In these cases, the hardware implementation 200 can be required to process the weights in a different manner, as described below.

[0121] In an example, the number of weights f can be too large such that not all filter weights can be stored in the coefficient buffer 230 at a given point in time. In this case, the hardware implementation 200 is configured to split the total number of weights f (each having size m*n*p) for a particular layer into two or more subsets of weights. The coefficient buffer can then store a first subset of the two or more subsets and pass the first subset to the convolution engines 240a-240n.

[0122] For example, the filter weights can be split into a first subset of weights and a second subset of weights such that the filter weights for a particular layer are in either the first subset or the second subset (in other examples, there can be any number of subsets). The first subset of weights can include a first number f1 of filter weights (i.e. ) can be stored in the coefficient buffer as will be described below. Once the first number f1 of filter weights have been processed to generate a first partial result, the result values can be stored in an accumulator until passed to the accumulation buffer 250 or can be passed to an external memory via the memory interface 210. Having stored the result values from processing the first number f1 of filter weights, the hardware implementation 200 can be configured to load a second subset of weights including a second number f2 of weights (i.e. ) into the coefficient buffer and process the second subset of weights as described above. f2 can be equal to f1 or can differ in value. The second subset of weights forms a second partial result of the convolution operation with the input data. The second partial result for the second subset of weights can then be combined with the values calculated from the first subset of weights in order to obtain the complete set of synthesized intermediate data for the particular layer.

[0123] By splitting the filter into two or more subsets by processing the subsets separately, processing is possible for layers in which the number of filter weights is too large to fit in the coefficient buffer. Thus, it is possible to provide a flexible implementation that can process a CNN having layers with a large number of filter weights associated therewith while maintaining a relatively small silicon area.

[0124] To process the two subsets separately, the memory interface 210 is configured to receive a first set of weights for a particular layer and the input data (or possibly a subset of the input data for the particular layer). As set out above, a first subset of the weights is stored in the coefficient buffer 230 and the input data for that layer is stored in the input buffer 235a-n. The first subset of the weights is used to process the input data and the values generated by processing the input data using the first subset of weights are stored back into the external memory via the memory interface 210. Subsequently, the input data for that layer is re-read from memory (e.g. the same data is read again) and stored in the input buffer 235a-n. In addition, a second subset of the weights for the layer is retrieved and stored in the coefficient buffer 230. The input data for that layer is then processed again using the second subset of weights in the same way as described above in relation to the first subset of weights. The values generated by processing the input data using the second subset of weights for that layer are then stored in the external memory.

[0125] Figure 6 An example method of processing according to the above described weights in two or more subsets is illustrated. At step 610, a first subset of the weights for a layer is received and processed at step 620 by convolving the first subset of weights with at least a portion of the input data in a convolution engine. The result of this convolution operation is a first partial result held in an accumulation buffer, as noted previously. Then, at step 630, a second subset of the weights for the same layer is received and processed at step 640 using the same convolution engine, resulting in the generation of a second partial result stored in the accumulation buffer. The first and second partial results are combined to form at least a combined portion of the result for the CNN layer.

[0126] Thus, the processing of the entire layer is completed in two separate stages. This process can be thought of as two or more passes through the same input data set, each pass using a different subset of the weights that together form the set of weights for the layer. Although two separate reads of the input data for a particular layer can be required, this is a useful trade-off as reduced silicon area is required to implement the hardware implementation described herein.

[0127] It will be appreciated that the number of subsets of weights that are generated and processed separately in different passes will depend on the size of the coefficient buffer and the number (and size) of the weights. It can be that two or more subsets of the weights need to be defined and processed separately.

[0128] Size of data

[0129] In another case, it can be necessary to split the input data into one or more subsets as a result of determining that the size of the input data is too large to be processed in a single pass. This case can occur, for example, in the case where the size of the input data to be processed in a particular layer is too large to be stored in the input buffer 235a-n. Therefore, it is necessary to implement splitting of the input data for a particular layer by hardware to process the data in more than two passes.

[0130] Figure 7a An example structure of the input data 760 is illustrated. The size of the input data is defined as a sequence of two-dimensional planes, and each plane is defined by two dimensions x x y and the input data has p planes for a layer. To form two subsets of data that are processed separately, the data region x x y x p can be split into more than two separate sub-regions, each defined as x1x y x p and x2x y x p, as illustrated in Figure 7a where the two defined sub-regions are separated by line 763. In this way, each of the subsets forms a portion of the input that can then be processed separately.

[0131] In particular, the first data sub-region is defined and processed in a similar manner to that previously described. Reference is made to Figure 7b the method illustrated. The first subset of input data (i.e. the first data sub-region) and the weight data are received at step 710 and passed to the convolution engines 240a to 240n. The first subset of input data is processed at step 720 using the convolution engines to generate a first partial result of the output values for the particular layer. Having processed the first subset of input data (i.e. the first data sub-region) and stored the result values back to the external memory, a subset of input data (i.e. the second data sub-region) is received at step 730 and loaded into the input buffer for processing in a similar manner to that used at step 740. For the second and subsequent subsets of data (sub-regions), it is also necessary to reload a portion of the first data sub-region to allow the convolution kernel to process the region of data near the boundary between the first sub-region and the second sub-region. An example of the data that is reloaded is illustrated in Figure 7a where the portion of the first sub-region defined between line 713 and line 715 is reloaded (e.g. from the external memory). The proportion of the first sub-region that is reloaded will depend on the size of the convolution kernel (m x n) and its relative position as it scans through the input data.

[0132] It will be appreciated that any number of sub-regions can be defined, and the particular number of sub-regions will depend on the size of the data and input buffer in a particular implementation.

[0133] Size of filter weights

[0134] Another case in which the input received by a hardware implementation of a CNN will be handled differently is where the size of a particular filter is too large to fit into the arrangement in the coefficient buffer. In order to split the single filter into two parts, referred to herein as sub-filters, the values of the intermediate calculations need to be stored in an accumulator buffer and the values recovered from the accumulator buffer, as will be described in more detail below.

[0135] In Figure 8a filter weights to be split into two sub-filters are illustrated. The filter weights 860 have dimensions m x n x p. As can be seen from Figure 8a In

[0136] In Figure 8b a method of processing the first and second sub-filters is illustrated. For the particular layer that the filter weights 860 will be used for, the first sub-filter 861 is received at step 810 and the input data for the layer is retrieved (e.g. from external memory). At step 820, the first sub-filter is passed to the convolution engine 240a-n and a convolution operation is performed in the convolution engine. The results of the convolution operation form the first partial result which is then passed to the accumulator buffer 250.

[0137] In an example, the first partial result resulting from the convolution of the first sub-filter 861 with the input data for the layer is passed from the accumulator buffer 250 to the external memory via the memory interface 210 for storage.

[0138] Having stored the first partial result of the convolution of the first sub-filter 861 with the input data for the layer (e.g. in memory), at step 830 of Figure 8b the input data is provided to the convolution engine along with the second sub-filter 862 (e.g. again from memory). The second sub-filter is then used to perform a convolution operation on the input data at step 840 to form a second partial result. Figure 8bThe input data is processed at step 840, where a corresponding convolution calculation is performed based on the same input data and a second sub-filter 862, and the result forms a second partial result that is passed to the accumulation buffer 250. The accumulation buffer then retrieves the intermediate result of the convolution of the first sub-filter 861 with the input data for that layer (e.g. from memory). The first partial result for the first sub-filter is then combined with the second partial result for the second sub-filter. The combined result is the result of applying the full filter to the input data. In some arrangements, further sub-filters can be used, in which case the combination of the results of the first and second sub-filters will form a further partial result, which will need to be combined with other partial results. In order to combine the results of different sub-filters to form the result output for a particular layer, the result of the first sub-filter is loaded into the accumulator(s) via the accumulation buffer 250 as the starting value of the accumulator of the output of the result of the second sub-filter.

[0139] It will be appreciated that the method described above can be extended to each filter in a particular layer. Also, it will be appreciated that the particular weights can be split into any number of sub-filters depending on the size of the filter weights and the size of the coefficient buffer.

[0140] Thus generally, the hardware implementation can be configured to divide or split at least one of the input data (for that layer) and the weight data (for that layer) into first and second (or more) parts and to process those parts separately to generate partial values. The hardware implementation can then be configured to combine the partial values together either by storing the values together in external memory or by numerically combining the partial values together in an accumulation buffer. It will be appreciated that other methods for combining the partial values together can be implemented.

[0141] In some arrangements, a combination of one or more of the methods described above can be used to split the received data into several smaller data sets, either for a single layer or across all layers. Some layers can not require any of the above splitting methods, where the size and number of filter weights are small enough to fit in the coefficient buffer at the same time and where the input data can fit into the input buffer.

[0142] In some examples of the hardware implementation 200, the command decoder 220 is configured to provide control of the hardware 200 in order to implement a CNN. As part of this control, the command decoder 220 can receive a series of commands that define the operations of the CNN. In some arrangements, the commands that define the operations of the CNN can be recovered from an external memory via the memory interface 210 and can precisely define which layers of the CNN require the data for that layer to be split into separate parts for processing. The received commands can also define the manner in which it is split into separate parts.

[0143] In addition, the format in which data is stored in the coefficient buffer and the input buffer can be defined in the received command, enabling the hardware implementation 200 to read data from the buffers. The performance of a hardware implementation in a CNN will depend at least in part on the number of elements of weight and input data that can be stored in a buffer at one time, as it is desirable to reduce the number of times memory read / writes occur. In another aspect of the application, processes of generating configuration data for use and reformatting input data in a hardware implementation of a CNN are described below.

[0144] The above arrangements discuss several different ways in which data to be processed in a convolution engine can be separated into smaller portions for processing. For example, input data can be split into sub-regions that are processed individually. Weight data can be split into separate weights that are processed separately or individual weights can be separated for processing. It will be understood that any combination of these portions can be implemented in a hardware implementation. Figure 2 For example, it is possible to provide only a sub-region of input data and a subset of weights to the convolution engine for processing during a single pass - the combination of which is Figure 6 , Figure 7a and Figure 7b features of the arrangements of Figure 6 , Figure 7a , Figure 7b , Figure 8a and Figure 8b described in Figure 6 , Figure 7a and Figure 7b may need to be performed multiple times in order to complete processing of a particular layer of a CNN using the hardware implementation 200. Multiple "passes" of the convolution engine are set out below with reference to Figures 9 to 12

[0145] Figure 9 A method 900 of processing data in a hardware implementation of a CNN is illustrated. In particular, Figure 9 The method of Figure 2 illustrates a method of processing all layers of a CNN using a hardware implementation such as Figure 9 The method 900 of Figure 5 arrangement in which data to be processed (either weights or input data or a combination thereof) is divided into multiple subsets that are each processed individually.

[0146] Figure 9 ​The method of FIG. 9 begins at step 905, where processing of the next layer is initiated. First, the next layer is the first layer in the CNN. It can also include receiving any command data that is common to the layer, such as the data format or weights to be used in processing the layer and control information that can be used to control the various hardware blocks and their interactions of a hardware implementation. For example, the command data can define whether the data in the accumulation buffer after each pass will be passed to memory or whether it will be reloaded into the input buffer. Also, the command data can also include data defining the activation function to be implemented in the activation module. The command data also defines the way in which the data will be divided into Figure 2 subsets for the layer. For example, the command data can define the number of passes required to process a particular layer. For example, the command data can define the number of passes to complete processing of a particular subset. The command data can also define the number of subsets of data that are defined collectively for the layer.

[0147] At step 910, a first subset of data is received. As previously described, the first subset of data can be received via a memory interface and can be stored in a coefficient buffer or an input data buffer for processing. The subset of data for the layer can include at least a portion of the weight data and / or at least a portion of the input data, such that more than one subset of data is required to process the data for the layer. At step 920, the first subset of data is processed in the convolution engine, for example, in the convolution engine of FIG. 8. Figure 2 At step 910, a first subset of data is received. As previously described, the first subset of data can be received via a memory interface and can be stored in a coefficient buffer or an input data buffer for processing. The subset of data for the layer can include at least a portion of the weight data and / or at least a portion of the input data, such that more than one subset of data is required to process the data for the layer. At step 920, the first subset of data is processed in the convolution engine, for example, in the convolution engine of FIG. 8.

[0148] If the subset convolution is not complete, the method returns to step 920 and the first subset of data is processed for further data. This process is repeated (e.g., using each pass of the convolution engine of FIG. 8) until the first subset of data has been processed for all of the required data, whether input data, weight data, or a combination thereof. Figure 2

[0149] ​Once complete, the method continues to step 940, where it is determined whether all subsets for the layer have been processed. In the first pass, not all subsets will have been processed in the case where only the first subset of data has been processed. In subsequent passes, it is determined whether all subsets for the layer have been processed. If not all subsets for the layer have been processed, the method returns to step 910, where a subsequent (e.g., second) subset of data for the layer is processed at step 920 (e.g., using the convolution engine of Figure 2 .

[0150] Once it has been determined that all subsets for the layer have been processed at step 940, it can be inferred that processing of the layer has been completed. At step 950, it is determined whether all layers of the CNN have been processed. If not all layers have been processed, the method returns to step 905, where processing of the next layer is initiated by loading command or control data for the layer. The processing described above is then repeated for each layer, and once complete, the CNN output data 960 is output.

[0151] Figure 10 Referring to Figure 6 , the arrangement and describe processing of layers of a CNN implemented in hardware.

[0152] Figure 10 The method 1000 begins at step 1005, where processing of the next layer is initiated. Initially, the next layer is the first layer in the CNN. This can include any command data that is generic to the layer, such as the data format or weights to be used in processing the layer and control information that can be used to control the various hardware blocks and their interactions of the hardware implementation. For example, the command data can define whether the data in the accumulation buffer after each pass will be passed to memory or whether it will be reloaded into the input buffer. Also, the command data can also include data defining the activation function to be implemented in the activation module.

[0153] At step 1010, a first subset of weights is received. As described previously, the first subset of weights can be received via the memory interface and can be stored in the coefficient buffer for processing. At step 1020, the first subset of weights can be processed for at least a portion of the input data, for example, in the convolution engine of Figure 2 . Then, at step 1030, it is determined whether all of the input data for the layer has been convolved with the first subset of weights. If not, the method returns to step 1020 and the first subset of weights is processed for further input data. This process is repeated (e.g., using each pass of the convolution engine of Figure 2 ) until the first subset of weights has been processed for all of the input data.

[0154] Once complete, the method proceeds to step 1040 where it is determined whether all weights for the layer have been processed. In the first pass, not all weights for the layer will have been processed in the case where only a first subset of the weights have been processed, as the first subset of weights is a subset of all the weights for the layer. If not all subset of weights for the layer have been processed, the method returns to step 1010 where a subsequent (e.g., second) subset of weights is processed for all input data for the layer at step 1020 (e.g., using the convolution engine of Figure 2 ).

[0155] Once it has been determined at step 1040 that all subset of weights for the layer have been processed, it can be inferred that processing of the layer has been completed. At step 1050, it is determined whether all layers of the CNN have been processed. If not all layers have been processed, processing returns to step 1005 where processing of the next layer is initiated by loading command or control data for the layer. The process described above is then repeated for each layer and CNN output data 1060 is output.

[0156] Figure 11 The arrangement of Figure 7b and describes processing of layers of a CNN implemented using hardware. The method is similar to the method of Figure 9 and Figure 10 In this arrangement, the subset of data comprises a subset of input data. Figure 11 The arrangement of Figure 9 and Figure 10 operates in a similar manner to the arrangement of Figure 11 In the arrangement of Figure 2 a subset of input data is processed using at least a portion of the weights at step 1120 (e.g., using the convolution engine of Figure 9 and Figure 10 This processing is repeated in a similar manner to Figure 7a and until all of the weights have been processed using the subset of input data for the layer. This process is repeated for all subsets for the layer and then for each layer of the CNN. As noted previously, due to the required overlap when using filters of a particular size, it can be necessary to reload a portion of a previously processed subset of input data, as illustrated in

[0157] Figure 12 Figures 9 to 11The method of FIG. 1 1 1 1, wherein the first subset comprises a first subfilter and the subsequent subsets comprise subsequent subfilters for the layer. Thus, the method comprises receiving one or more subfilters for the current layer at step 1210 and then processing at least a portion of the input data for the layer using the one or more subfilters at step 1220. At step 1230, it is determined whether all of the input data for the layer has been processed using the one or more subfilters, and if not, the method returns to step 1220 to process at least another portion of the input data using the one or more subfilters. The process is repeated for each subfilter of the subfilters and the results are combined (e.g., in an accumulation buffer of the hardware arrangement or accumulator). Figure 2

[0158] Figure 13 A computer system in which a hardware implementation of a CNN or activation module for a CNN as described herein can be implemented is shown. The computer system includes a CPU 1302, a GPU 1304, a memory 1306, and other devices 1314 such as a display 1316, speakers 1318, and a camera 1317. The processing block 1310 is implemented on the GPU 1304. In other examples, the processing block 1310 can be implemented on the CPU 1302. The processing block 1310 can also be implemented in a separate unit (not shown). The processing block 1310 can include digital logic configured to implement the hardware implementation 200. The hardware implementation 200 can be controlled by an external processor such as the GPU 1304 or the CPU 1302 or another processor not shown. The components of the computer system can communicate with each other via a communication bus 1320. Storage 1312 is implemented as part of the memory 1306.

[0159] Figures 1 to 12 The hardware implementation 200 of a CNN or activation module for a CNN as illustrated in FIG. 13 is shown to comprise a number of functional blocks. This is merely schematic and is not intended to define strict divisions between different logical elements of such entities. Each functional block can be provided in any suitable way. It will be understood that the intermediate values described herein as formed by the CNN or activation module for a CNN hardware implementation need not be physically generated at any time by the CNN or activation module for a CNN hardware implementation, and can merely represent a convenient way of describing the processing performed by the CNN or activation module for a CNN hardware implementation between its inputs and outputs.

[0160] ​The hardware implementations of the CNNs or activation modules for CNNs described herein can be implemented in hardware on an integrated circuit. The hardware implementations of the CNNs or activation modules for CNNs described herein can be configured to perform any of the methods described herein. Generally, any of the above described functions, methods, techniques, or components can be implemented in software, firmware, hardware (fixed logic circuitry), or any combination thereof. The terms "module," "functionality," "component," "element," "unit," "block," and "logic" are used herein to generally refer to software, firmware, hardware, or any combination thereof. In the case of a software implementation, the module, functionality, component, element, unit, block, or logic represents program code that performs specified tasks when executed on a processor. The algorithms and methods described herein can be performed by one or more processors executing code to cause the processor(s) to perform the algorithms / methods. Examples of computer-readable storage media include random access memory (RAM), read only memory (ROM), optical disc, flash memory, hard disk storage, and other storage devices that can use magnetic, optical, and other technologies to store instructions or other data that can be accessed by a machine.

[0161] The terms computer program code and computer readable instructions as used herein refer to any kind of executable code for a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code that defines integrated circuits such as hardware description languages or netlists, and code expressed in a programming language such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that when suitably executed, processed, interpreted, compiled, executed in a virtual machine or other software environment, causes a processor of the computer system in which the executable code is supported to perform the tasks specified by the code.

[0162] A processor, computer, or computer system can be any kind of device, machine, special-purpose circuit, or collection or part thereof having processing capability such that it can execute instructions. A processor can be any kind of general- or special-purpose processor such as a CPU, GPU, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system can include one or more processors.

[0163] It is also intended to encompass software (such as HLD (hardware description language) software) that defines the configuration of hardware as described herein, such as software that is used to design integrated circuits or to configure programmable chips to perform desired functions. That is, computer readable program code that has been encoded onto a computer readable storage medium in the form of an integrated circuit definition data set, which when processed in an integrated circuit manufacturing system, configures the system to manufacture a hardware implementation of a CNN or activation module for a CNN configured to perform any of the methods described herein, or to manufacture a hardware implementation of a CNN or activation module for a CNN comprising any of the apparatus described herein. The integrated circuit definition data set can be, for example, an integrated circuit description.

[0164] Thus, there can be provided a method of manufacturing a hardware implementation of a CNN or activation module for a CNN at an integrated circuit manufacturing system, as described herein. Also, there can be provided an integrated circuit definition data set which, when processed in an integrated circuit manufacturing system, causes a method of manufacturing a hardware implementation of a CNN or activation module for a CNN to be performed.

[0165] The integrated circuit definition data set can be in the form of computer code (for example, as a netlist, code for configuring a programmable chip, as a hardware description language defining an integrated circuit at any level, including register transfer level (RTL) code, as a high level circuit representation (Verilog or VHDL) and as a low level circuit representation (such as OASIS (RTM) and GDSII)). A high level representation of an integrated circuit (such as RTL) which logically defines the integrated circuit can be processed at a computer system configured for generating a manufacturing definition of the integrated circuit in the context of a software environment comprising definitions of circuit elements and rules for combining those elements in order to generate a manufacturing definition representing an integrated circuit so defined. As is usual with software executed at a computer system in order to define a machine, one or more intermediate user steps (for example, providing commands, variables etc.) can be required for the computer system configured for generating a manufacturing definition of an integrated circuit to execute the code defining the integrated circuit in order to generate a manufacturing definition of that integrated circuit.

[0166] Examples will now be described with respect to Figure 14 processing an integrated circuit definition data set at an integrated circuit manufacturing system in order to configure the system to manufacture a hardware implementation of a CNN or activation module for a CNN.

[0167] Figure 14An example of an integrated circuit (IC) manufacturing system 1402 is shown, which includes a layout processing system 1404 and an integrated circuit generation system 1406. The IC manufacturing system 1402 is configured to receive an IC definition data set (e.g., defining a CNN or a hardware implementation of an activation module for a CNN, as described in any of the examples herein), process the IC definition data set, generate an IC from the IC definition data set (e.g., implementing a CNN or a hardware implementation of an activation module for a CNN, as described in any of the examples herein). The processing of the IC definition data set configures the IC manufacturing system 1402 to manufacture an integrated circuit implementing a CNN or a hardware implementation of an activation module for a CNN, as described in any of the examples herein.

[0168] The layout processing system 1404 is configured to receive and process the IC definition data set to determine a circuit layout. Methods of determining a circuit layout from an IC definition data set are known in the art and can include, for example, synthesizing RTL code to determine a gate level representation of the circuit to be generated (e.g., in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components)). The circuit layout can be determined from the gate level representation of the circuit by determining location information for the logic components. This can be done automatically or with user involvement in order to optimize the circuit layout. When the layout processing system 1404 has determined the circuit layout, it can output a circuit layout definition to the IC generation system 1406. The circuit layout definition can be, for example, a circuit layout description.

[0169] The IC generation system 1406 generates an IC from the circuit layout definition, as is known in the art. For example, the IC generation system 1406 can implement a semiconductor device manufacturing process to generate the IC, which can include a multi-step sequence of photolithographic and chemical processing steps during which electronic circuits are progressively created on a wafer made of semiconductor material. The circuit layout definition can be in the form of a mask that can be used in a photolithographic process for generating an IC from a circuit definition. Alternatively, the circuit layout definition provided to the IC generation system 1406 can be in the form of computer readable code that the IC generation system 1406 can use to form appropriate masks for use in generating an IC.

[0170] The different processes performed by the IC fabrication system 1402 can all be implemented in one location (e.g., by one party). Alternatively, the IC fabrication system 1402 can be a distributed system such that some of the processes can be performed at different locations and can be performed by different parties. For example, some of the stages: (i) synthesizing RTL code representing the IC definition data set to form a gate level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate level representation; (iii) forming a mask from the circuit layout; and (iv) fabricating an integrated circuit using the mask can be performed at different locations and / or by different parties.

[0171] In other examples, processing of an integrated circuit definition data set in an integrated circuit fabrication system can configure the system to fabricate a CNN or a hardware implementation of an activation module for a CNN without the IC definition data set being processed in order to determine a circuit layout. For example, the integrated circuit definition data set can define a configuration of a reconfigurable processor such as an FPGA, and processing of the data set can configure the IC fabrication system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data to the FPGA).

[0172] In some embodiments, when processed in an integrated circuit fabrication system, an integrated circuit fabrication definition data set can cause the integrated circuit fabrication system to generate a device as described herein. For example, by an integrated circuit fabrication definition data set to configure the integrated circuit fabrication system to generate a device as described herein with respect to Figure 14 Configuration of an integrated circuit fabrication system in the manner described above can cause a device as described herein to be fabricated.

[0173] In some examples, an integrated circuit definition data set can include software running on or in conjunction with hardware defined at the data set. In Figure 14 In the example shown in FIG. 1, the IC generation system can also be configured by the integrated circuit definition data set to, when fabricating an integrated circuit, load firmware onto the integrated circuit or otherwise provide the integrated circuit with program code defined at the integrated circuit definition data set for use with the integrated circuit.

[0174] The Applicant hereby discloses by way of disclaimer any subject matter that is not described or covered by at least one of the following aspects of the application: (i) each individual feature described herein; and (ii) any combination of two or more such features, to the extent such features or combinations are capable of being performed according to the present specification as a whole, whether or not the subject matter addressed by any of the foregoing aspects solves any problem disclosed herein. In light of the foregoing description, it will be evident to a person of ordinary skill in the art that various modifications can be made within the scope of the application.

Claims

1. Hardware for implementing a convolutional neural network (CNN), the hardware comprising: a memory interface configured to receive, from an external memory, weights and input data to be used in computations within the CNN and command information for controlling operation of the hardware; a coefficient buffer controller configured to receive the weights from the memory interface and pass the weights to a coefficient buffer; the coefficient buffer configured to store the weights received from the coefficient buffer controller; an input buffer controller configured to receive the input data from the memory interface and pass the input data to a plurality of input buffers; the plurality of input buffers configured to store the input data received from the input buffer controller; a command decoder configured to decode the command information and subsequently issue control information to the coefficient buffer controller and the input buffer controller to control the manner in which the weights and the input data are stored in the coefficient buffer and the plurality of input buffers, respectively; a plurality of convolution engines configured to perform one or more convolution operations on the input data in the plurality of input buffers using the weights in the coefficient buffer; a plurality of accumulators configured to receive a result output of the plurality of convolution engines and add the result output of the plurality of convolution engines to a value stored in an accumulation buffer; the accumulation buffer configured to store an accumulated result from the plurality of accumulators; a shared buffer; an activation module configured to perform at least one of a plurality of different activation functions on data in the accumulation buffer and store a resulting value in the shared buffer; a normalization module configured to perform one of a plurality of different normalization functions on data in the shared buffer and store a result in the shared buffer; and a pooling module configured to perform a pooling operation on data in the shared buffer and store a result in the shared buffer. the activation module is configured to:

2. The hardware of claim 1, wherein, receive configuration data specifying one or more properties of an activation function to be applied to the input data; store data representing the activation function in entries of a lookup table based on the received configuration data; receive input data representing an input to the activation function and read one or more entries from the lookup table based on at least a portion of the received input; and calculate an output value representing an output of the activation function based on the one or more entries read from the lookup table. the configuration data specifies one or more of: one or more data points of the activation function, one or more characteristics of the activation function, and one or more values that can be used to derive or identify the activation function. ​ 3. The hardware of claim 2, wherein, ​ 4. The hardware of claim 2, wherein, The data representing the activation function comprises the received configuration data.

5. The hardware of claim 2, wherein, The activation module uses the configuration data to identify a pre-stored activation function, and the data representing the activation function represents the identified pre-stored activation function.

6. The hardware of claim 2, wherein, (i) the activation module is configured to use at least part of the received input data as an address to the lookup table, (ii) the activation module is configured to compute the output value also based on at least part of the input data, and / or (iii) the activation module is configured to compute the output value by interpolating between two or more entries read from the lookup table.

7. The hardware of claim 1, wherein, The activation module comprises: a lookup table configured to store values representing the activation function performed by the activation module, each entry of the lookup table comprising a gradient and an offset; multiplication logic configured to multiply a part of the input data with a gradient retrieved from the lookup table; and addition logic configured to add an offset retrieved from the lookup table to an output of the multiplication logic; wherein the gradient and the offset are retrieved from the lookup table based on another part of the input data.

8. The hardware of claim 1, wherein, The activation module comprises: a lookup table configured to store values representing the activation function performed by the activation module; and interpolation logic configured to interpolate between at least two values retrieved from the lookup table using a part of the input data; wherein the at least two values are retrieved from the lookup table based on another part of the input data.

9. The hardware of any one of claims 1 to 8, wherein, At least one of the activation module, the normalization module and the pooling module is configured to receive control signaling configuring the at least one of the activation module, the normalization module and the pooling module into a bypass mode in which the at least one of the activation module, the normalization module and the pooling module does not perform its respective function and the data value simply passes through the at least one of the activation module, the normalization module and the pooling module without change.

10. The hardware of any one of claims 1 to 8, wherein, The memory interface is configured to pass data from the external memory directly to the activation module via the accumulation buffer.

11. The hardware of any of claims 1 to 8, further comprising a plurality of multiplexers connected to each of the plurality of input buffers, each multiplexer configured to select an output from one of the plurality of input buffers and pass a value output from the selected input buffer to a respective convolution engine of the plurality of convolution engines.

12. The hardware of any of claims 1 to 8, wherein: an output of each of the plurality of convolution engines is provided as input to a respective accumulator of the plurality of accumulators; and the accumulation buffer is connected to an input of each of the plurality of accumulators and is configured to feed back values to the plurality of accumulators to enable the plurality of accumulators to perform accumulation computations.

13. The hardware of any one of claims 1 to 8, wherein, The accumulation buffer is connected to the memory interface and is configured to send data to and receive data from the external memory via the memory interface.

14. The hardware of any one of claims 1 to 8, wherein, The standardization module is configured to receive signaling to indicate which of the plurality of different standardization functions the standardization module is to perform.

15. The hardware of any one of claims 1 to 8, wherein, The standardization module is configured to receive one or more parameters of the standardization function performed by the standardization module, the one or more parameters relating to one or more layers of the CNN.

16. The hardware of any one of claims 1 to 8, wherein, The command information comprises information relating to the size and / or format of the weight data, the size and format of the input data, and their location in the external memory.

17. The hardware of any one of claims 1 to 8, wherein, The shared buffer is connected to the memory interface and the memory interface is configured to transfer data stored in the shared buffer to the external memory or the input buffer controller.

18. The hardware of any one of claims 1 to 8, wherein, The hardware is implemented on an integrated circuit.

19. The hardware of any one of claims 1 to 8, wherein, The shared buffer is internal to the hardware and distinct from the external memory, and / or the standardization module is distinct from the activation module.

20. A non-transitory computer readable storage medium having stored thereon computer readable descriptions of an integrated circuit, the computer readable descriptions, when processed in an integrated circuit manufacturing system, cause the integrated circuit manufacturing system to manufacture a hardware according to any one of claims 1 to 19.

Images