Activation functions for deep neural networks

Abstract

The present application discloses an activation function for a deep neural network. A method for configuring hardware for implementing a deep neural network (DNN) to perform an activation function, the hardware including a programmable lookup table for storing lookup data at an activation module for performing the activation function, the lookup data approximating the activation function over a first range of input values ​​to the activation module, the method comprising: providing calibration data to a representation of the hardware; monitoring input to the activation module of the representation of the hardware to determine a range of input values ​​to the activation module; generating lookup data for the lookup table, the lookup data representing the activation function over the determined range of input values; and loading the generated lookup data into the lookup table of the hardware, thereby configuring the activation module of the hardware to perform the activation function over the determined range of input values.

Description

Technical Field

[0001] The present disclosure relates to methods of configuring hardware for implementing a deep neural network. Background Art

[0002] A deep neural network (DNN) is an artificial neural network with multiple layers between the input layer and the output layer. DNN can be used in machine learning applications. In particular, deep neural networks can be used in signal processing applications, including image processing and computer vision applications.

[0003] DNNs are often implemented in applications where power resources are not a significant factor. Nevertheless, DNNs also have applications in some other technical fields where the resources of the hardware used to implement the DNN are such that power consumption, processing power, or silicon area are limited. Furthermore, the definition of a DNN for a particular application may change over time—for example, due to more training of the DNN.

[0004] There is therefore a need for a system to efficiently implement DNNs in an area- and power-efficient manner that is flexible to changes in the definition of the DNN. Summary of the invention

[0005] This section is provided to introduce some of the concepts that are further described below. This section should not be considered to identify key features or essential features of the claimed subject matter, nor should it be used to limit the scope of the claimed subject matter.

[0006] A method of configuring hardware for implementing a deep neural network (DNN) to perform an activation function is provided, the hardware comprising a programmable lookup table at an activation module for performing the activation function to store lookup data, the lookup data approximates the activation function over a first range of input values ​​to the activation module, the method comprising:

[0007] Providing calibration data to the representation of the hardware;

[0008] monitoring an input to an activation module of the hardware representation to determine a range of input values ​​to the activation module;

[0009] generating lookup data for a lookup table, the lookup data representing the activation function over the determined range of input values; and

[0010] The generated lookup data is loaded into a lookup table of the hardware, thereby configuring an activation module of the hardware to perform the activation function over the determined range of input values.

[0011] The method may further include:

[0012] receiving an input data stream to the hardware; and

[0013] The activation function is performed over the determined range of input values ​​using a lookup table.

[0014] The DNN may use an activation module to execute multiple activation functions to process an input data stream, and the method may also include: repeating these providing, monitoring, generating and loading steps for each activation function to generate and load lookup data for a lookup table, which looks up data represents each of the multiple activation functions.

[0015] The lookup table may include two sets of lookup data and be able to switch between the two sets of lookup data. When the activation module executes a series of activation functions, the first activation function in the series and the loading of the generated lookup data for the next activation function in the series into the lookup table may be executed simultaneously.

[0016] The method may also include: upon receiving an input data stream to the hardware, checking whether the input value to the activation module falls outside the determined range of input values, and if the input value to the activation module falls outside the determined range of input values, using the activation function value corresponding to the closest extreme value of the determined range of input values ​​as the output value of the activation function.

[0017] Monitoring the input to the activation module may also include determining an offset that, when subtracted from each input value to the activation module, causes the range of input values ​​to be substantially centered around a predetermined input value, and executing the activation function may further include subtracting the offset from each input value received at the activation module before looking up each input value in a lookup table.

[0018] Executing the activation function may include, upon receiving a first input value, looking up in a lookup table a pair of adjacent data points closest to the first input value, and interpolating between a pair of corresponding values ​​of the activation function to form a value estimate of the activation function corresponding to the first input value.

[0019] A predetermined number of most significant bits of the first input value may be used as lookup addresses in the lookup table, with the remaining bits of the first input value being used in interpolating between the pair of corresponding values ​​of the activation function.

[0020] The lookup table may include a first database and a second database (data store), the first database including a first set of data points, the second database including a second set of data points, such that for each adjacent pair of data points, one of the data points is in the first database and the other data point is in the second database, and performing the activation function for the first input value includes: simultaneously searching for each of the pair of adjacent points in the first or second database of each of the pair of adjacent points.

[0021] The interpolation may be linear interpolation.

[0022] Generating the lookup data may include computing a set of curves that approximate the activation function over the determined range of input values, each curve representing a portion of the activation function such that the set of curves together identify an output value for each input value within the determined range.

[0023] The method may also include: upon receiving an input data stream to the hardware, checking whether an input value to the activation module falls outside a determined range of input values, and if the input value to the activation module falls outside the determined range of input values, extrapolating the closest curve in the set of curves to provide an output value of the activation function.

[0024] The curves in the set of curves may be linear or quadratic.

[0025] Generating the lookup data may include computing a set of data points representing an activation function over the determined range of input values.

[0026] Monitoring inputs to the activation module may include identifying maximum and minimum input values ​​to the activation module and using the maximum and minimum input values ​​as boundaries for a determined range of input values.

[0027] Monitoring inputs to an activation module may include forming a histogram of input values ​​representing the probability of occurrence of the input values, and using a pair of input values ​​between which a predetermined or programmable portion of the input value distribution falls as boundaries of a determined range of input values.

[0028] The calibration data may include exemplary input data selected to represent various possible inputs to the hardware.

[0029] The determined range of input values ​​may be smaller than the range of possible input values ​​based on the bit length of the input values ​​over which the lookup data represents the activation function.

[0030] The lookup data may also represent the activation function over a range equal to the determined range of input values.

[0031] The number of entries in the lookup data representing an activation function over the determined range of input values ​​may be equal to the number of entries in the lookup table for that activation function.

[0032] The method may be performed following optimization of the DNN.

[0033] According to a second aspect, there is provided a data processing system, comprising:

[0034] A hardware representation for implementing a deep neural network (DNN), the hardware representation comprising an activation module for executing an activation function, the activation module comprising a programmable lookup table for storing lookup data representing the activation function; and

[0035] Configuration modules are configured to:

[0036] enabling calibration data to be provided to the hardware in a representation form;

[0037] monitoring an input to an activation module of the expression to determine a range of input values ​​to the activation module; and

[0038] generating lookup data for a lookup table, the lookup data representing the activation function over the determined range of input values;

[0039] Therein, when loaded into a lookup table of an activation module of the hardware, the generated lookup data is suitable for causing the activation module to execute the activation function over a determined range of input values.

[0040] Processing of the DNN may require execution of multiple activation functions, and the configuration module is configured to determine a range of input values ​​to the activation module for each activation function and generate corresponding lookup data representing each activation function.

[0041] The hardware may include a plurality of activation modules, the configuration module being configured to independently generate lookup data for each activation function executed at each activation module.

[0042] The configuration module may be provided in the form of software running on a data processing system.

[0043] According to a third aspect, there is provided hardware for implementing a deep neural network (DNN), comprising an activation module for performing an activation function, the activation module having a programmable lookup table for storing lookup data representing the activation function, wherein in use, the activation module is configured to load first lookup data into the lookup table, the first lookup data being generated over a determined range of input values ​​to the activation module for performing the activation function;

[0044] Wherein the range of input values ​​to the activation module is determined by monitoring the input to the activation module in the hardware representation, the activation module being arranged to operate on calibration data.

[0045] According to a fourth aspect, there is provided hardware for implementing a deep neural network (DNN), comprising an activation module for executing an activation function, the activation module comprising:

[0046] a programmable lookup table for storing lookup data representing an activation function over a range of input values ​​to the activation module;

[0047] A rectified linear unit (ReLU), capable of performing a ReLU activation function; and

[0048] The configuration logic is configured to make the activation module perform the activation function at the ReLU and close the programmable lookup table when the DNN requires the activation module to implement the ReLU activation function.

[0049] When the DNN requires the activation module to implement the activation function using a lookup table, the ReLU unit can be configured to clamp the input value received at the activation module that falls outside the determined range of input values ​​to the closest extreme value of the determined range of input values, and the clamped input value is then passed to the lookup table to implement the activation function.

[0050] The data processing system may be configured to perform any of the methods disclosed herein.

[0051] The hardware disclosed herein may be implemented on an integrated circuit.

[0052] A method of manufacturing hardware using an integrated circuit manufacturing system is also provided.

[0053] A method of manufacturing the hardware described herein using an integrated circuit manufacturing system is provided, the method comprising:

[0054] processing the computer-readable description of the hardware using a layout processing system to generate a circuit layout description of an integrated circuit implementing the hardware; and

[0055] The hardware is manufactured from the circuit layout description using an integrated circuit generation system.

[0056] The computer program code may be adapted to perform any of the methods disclosed herein.

[0057] A non-transitory computer-readable storage medium having computer-readable instructions stored thereon is provided, which when executed at a computer system causes the computer system to perform any of the methods disclosed herein.

[0058] Also provided is an integrated circuit definition data set which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the hardware described herein.

[0059] Also provided is a non-transitory computer-readable storage medium having stored thereon a computer-readable description of the hardware described herein, which when processed in an integrated circuit manufacturing system causes the integrated circuit manufacturing system to:

[0060] processing the computer-readable description of the hardware using a layout processing system to generate a circuit layout description of an integrated circuit that implements the hardware; and

[0061] The hardware is manufactured from the circuit layout description using an integrated circuit generation system.

[0062] Also provided is an integrated circuit manufacturing system configured to manufacture the hardware described herein.

[0063] An integrated circuit manufacturing system is provided, comprising:

[0064] a non-transitory computer-readable storage medium having stored thereon a computer-readable description of the hardware described herein;

[0065] a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit implementing the hardware; and

[0066] An integrated circuit production system is configured to manufacture the hardware according to the circuit layout description. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present invention will be described by way of example with reference to the accompanying drawings. In the drawings:

[0068] Figure 1 An example of the structure of data used in a convolutional neural network is shown;

[0069] Figure 2 An exemplary hardware implementation of a convolutional neural network is shown;

[0070] Figure 3 Shows Figure 2 The lookup table used in the activation module shown;

[0071] Figure 4a An exemplary method for operating an activation module is shown;

[0072] Figure 4b shows a rectified linear unit (ReLU) function that can be implemented in an activation module;

[0073] Figure 4c shows a parameterized rectified linear unit (PReLU) function that can be implemented in an activation module;

[0074] Figure 4d Shows Figure 2 An exemplary hardware implementation of the activation module shown;

[0075] Figure 4e Shows Figure 4f An example of interpolation performed in the activation module of ;

[0076] Figure 4f Shows Figure 2 Another exemplary hardware implementation of the activation module shown;

[0077] Figure 4g Shows Figure 2 Another exemplary hardware implementation of the activation module shown;

[0078] Figure 5a The sigmoid activation function that can be implemented in the activation module is shown;

[0079] Figure 5b shows the hyperbolic tangent (tanh) activation function that can be implemented in the activation module;

[0080] Figure 6 is a schematic diagram of a data processing system, which includes Figure 2 Hardware implementation of

[0081] Figure 7 The tanh activation function is shown, for which the expected range of input values ​​has been determined;

[0082] Figure 8 The flowchart shown illustrates a method for performing an activation function in a hardware implementation of a deep neural network;

[0083] Fig. 9 It is a schematic diagram of the integrated circuit manufacturing system. DETAILED DESCRIPTION

[0084] The following description is set forth by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art. The embodiments are described by way of example only.

[0085] In the examples provided herein, the present invention is described as being implemented in a convolutional neural network (CNN). A convolutional neural network is a deep neural network (DNN) in which convolution operations are applied at one or more layers of the network. It will be appreciated that the present invention is not limited to use in a convolutional neural network, but can be used in any type of deep neural network.

[0086] Figure 1 An exemplary overview of the format of the data used in CNN is shown. Figure 1 It can be seen that the format of the data used in CNN can be formed by multiple planes. The input data can be arranged into P data planes, where each plane has dimensions x×y. CNN includes multiple layers, each layer with multiple filters w0…w n These filters w0…w n Each has dimensions m×n×p and is applied to the input data along directions s and t in several steps according to the convolution operation, as Figure 1 shown.

[0087] As mentioned above, each layer can be associated with multiple filters w0…w n The term "weight" as used herein may also refer to filters, filter weights, or coefficients. The number and value of filter weights may differ between layers, so for the first layer, the number of weights may be defined as For the second layer, the weight numbers can be defined as The number of weights in the first layer is n1, and the number of weights in the second layer is n2.

[0088] For multiple layers of a CNN, the input data for the layer is processed by convolving the input data of the layer using the weights associated with the layer. For the first layer, the "input data" can be regarded as the initial input to the CNN, which can be an image in some examples - for example, when the CNN is used for visual applications. The first layer processes the input data and generates a first set of intermediate data that is passed to the second layer. The first set of intermediate data can also take the form of multiple planes of data. The first set of intermediate data can be regarded as forming input data for the second layer, and the second layer processes the first intermediate data to generate output data in the form of second intermediate data. In the case where the CNN includes a third layer, the third layer receives the second intermediate data as input data and processes the data to generate output data in the form of second intermediate data. Therefore, references to "input data" herein can be interpreted as including references to input data of any layer. For example, the term "input data" can refer to intermediate data that is the output of a particular layer and the input of a subsequent layer. This situation is repeated until the last layer generates output data, which can be regarded as the output of the CNN.

[0089] Figure 2 An exemplary hardware implementation 200 configured to implement a CNN is illustrated. Figure 2 Only one example of hardware for the invention is shown: in general, the invention can be used with any hardware configuration suitable for implementing a CNN (or more generally, any kind of deep neural network).

[0090] Figure 2 The hardware implementation shown includes a digital logic circuit configured to receive data (e.g., configuration data) that defines the CNN to be implemented, and the circuit is also configured to operate according to the data definition to process input data to implement the CNN. Therefore, the hardware implementation can be regarded as configurable hardware for implementing one or more CNNs. Thus, the reference to the "hardware implementation" of a CNN herein is a reference to hardware configured to operate on input data according to the received configuration data to implement the CNN. The hardware implementation itself is not necessarily configured to implement a specific CNN, nor is it necessarily pre-loaded with specific configuration data for the CNN. For example, a CNN does not necessarily include pre-loaded (or hard-coded) weight data and layer definitions. Therefore, the hardware implementation is configured to implement any CNN based on the configuration data it receives. Thus, the configuration data (also referred to as "command data" herein) defines the specific CNN to be implemented, including the number of layers to be included and the size and value of the weight data, as well as the expected format of the input data.

[0091] The data defining the CNN and how to process the CNN may include configuration (or command) data, weight data, and input data. The hardware implementation is therefore configured to process the input data using the weight data to perform the operations defined by the CNN. The hardware implementation 200 includes a memory interface 210, an input buffer controller 215, a command decoder 220, a coefficient buffer controller 225, a coefficient buffer 230, n convolution engines 240, n accumulators 245, an activation module 255, a normalization module 265, a shared buffer 270, and a pooling module 275.

[0092] The memory interface 210 is configured to provide an interface between the hardware implementation 200 and an external memory (not shown). The external memory can be viewed as a module separate from the hardware implementation 200, or can be viewed as integrated with the hardware 200. The command or configuration data may include, for example, information about the size and / or format of the weight data, the size and format of the input data, and their location in the external memory.

[0093] The memory interface 210 is configured to receive weights and data to be used in calculations within the CNN from an external memory, as well as command information to control the operation of the hardware implementation 200. The received weights (also referred to herein as "coefficients") are passed to the coefficient buffer controller 225, and the received data is passed to the input buffer controller 215. The received commands are passed to the command decoder 220, which is configured to decode the commands and then issue control information to the elements of the hardware implementation to control the storage of weights and input data in the buffers, including the coefficient buffer controller 225 and the input buffer controller 215.

[0094] The weights and input data received from the external memory via the memory interface 210 during a read from the external memory may form weights and input data for only a portion of a layer, all weights and input data to be used in processing a layer, or may include weights and input data for processing multiple layers. For example, the weights received from the external memory may form the weights of a layer, and the received input data may form only a portion of the input data for a layer (or vice versa). Any combination of data and weights for one or more layers may be received from the external memory. Any number of reads may be used to read the weights and data from the memory. The weights and data may be read from the memory into the corresponding buffers 230 and 235 at the beginning of a hardware pass. The reads from the memory may be burst reads.

[0095] In practice, the maximum number of weights and data values ​​received from external memory may depend on the size of coefficient buffer 230 and input buffer 235, respectively. For a given hardware pass, the number of filters operating on a window (and therefore the number of weights and data values ​​to be maintained at buffers 230 and 235) may depend on how the processing of the various network layers is divided among multiple hardware passes. These weights are passed from coefficient buffer controller 225 to coefficient buffer 230, and the received data is passed from input buffer controller 215 to multiple input buffers 235a-235m. Figure 2 These input buffers are schematically shown in , and may be provided in any suitable manner, including as a single input buffer module that includes multiple memories (e.g., RAM banks) arranged to provide the m sets of input data. The number of input buffers will depend on the specific implementation of hardware 200 and may take any value. The number of input buffers 235 may be different from the number of convolution engines 240 - for example, where multiple convolution engines are arranged to operate on the same window of data, fewer input buffers may be provided than the number of convolution engines.

[0096] Input buffers 235a-235m are connected to each of the multiple multiplexers because each convolution engine 240a-240n needs to access all valid "strips" of input data. In some examples, n=m, so the number of input buffers is the same as the number of convolution engines, but in other examples, n≠m. In general, the number of input buffers 235 can be changed independently of the number of convolution engines 240. Each multiplexer is configured to select the output from one of these input buffers and pass the value output from the selected input buffer to the corresponding convolution engine 240a-240n. In addition, the weight from the coefficient buffer 230 is provided as a second input to each convolution engine 240a-240n. These convolution engines are configured to perform a convolution operation on the received input data using the weights received from the coefficient buffer 230. The output obtained by each convolution engine 240a-240n is provided to the corresponding accumulator in the multiple accumulators 245a-245n as input. Figure 3 An exemplary implementation of the convolution engine 240 n is illustrated, including a plurality of multiplication logic elements 310 and a plurality of addition logic elements 320 .

[0097] Each accumulator 245a-245n is connected to an accumulation buffer 250 ( Figure 2 A single accumulation buffer is shown, but in general any number of buffers may be provided at any number of databases. Accumulation buffer 250 is configured to store the accumulation results received from each accumulator 245a-245n. In some cases, the accumulator may provide a partial accumulation rather than the final result of the convolution operation. For example, where the filter is larger than the coefficient buffer, the convolution operation (including windowing the filter) may be divided into multiple hardware passes. The partial accumulation may be passed forward for use in a subsequent hardware pass so that the final accumulation result may be formed over multiple passes.

[0098] The accumulation buffer 250 may be connected to the memory interface 210. The accumulation buffer 250 may be configured to send data to and receive data from the external memory via the memory interface 210. The accumulation buffer 250 may be configured to be able to store values ​​at the external memory and read them back via the memory interface 210, as will be described in more detail below. The accumulation buffer 250 may be connected to the inputs of the accumulators 245a-245n so that partial accumulations can be fed back into the accumulators 245a-245n to be accumulated with subsequent partial accumulations performed at the convolution engines.

[0099] The accumulation buffer 250 is configured to pass the accumulated value to the activation module 255. The activation module 255 is configured to perform at least one of a plurality of different activation functions. The activation function is performed in all types of deep neural networks. Figure 2 In the example shown, the activation function performed by the activation module 255 forms part of the implementation of the CNN.

[0100] In use, when the CNN is implemented at the hardware implementation 200, the activation module is configured to perform the following Figure 4a The method shown. The activation module can be configured to implement one or more user-defined activation functions. The activation module can be configured to receive configuration data at step 401, which specifies one or more properties of the activation function to be applied to the input data. For example, the configuration data can be used to specify one or more data points of the activation function, one or more characteristics of the activation function, or one or more values ​​that can be used to derive or identify a specific activation function to be implemented in the activation module 255 for one or more passes of the activation function.

[0101] During configuration, the activation module 255 may be configured to store data representing an activation function in an entry of a lookup table based on the received configuration data at step 402. In some examples, the data representing the activation function may be the received configuration data. In some arrangements, if the configuration data includes data points of the activation function, these data points themselves may be stored in the lookup table. In some arrangements, the configuration data may be used to identify a pre-stored activation function, so that data representing the identified activation function (e.g., data points representing this pre-stored activation function) may be loaded into the lookup table (e.g., using an identifier in the configuration data). The values ​​in the lookup table may represent an estimate or approximation of the activation function. The data stored in the lookup table may be a gradient and offset value of at least a portion of the activation function.

[0102] In this way, the activation module can be configured to implement the appropriate activation function for the activation layer of the CNN for a given hardware pass. Since the lookup table is programmable, the activation function can change from one hardware pass to the next.

[0103] In use, after the configuration of the activation module at step 402, in the CNN, the activation module 255 receives input data at step 403, which represents the input to the activation function. The activation module can be configured to read one or more entries from the lookup table based on at least a portion of the received input. In some arrangements, the activation module 255 is configured to use at least a portion of the received input as an address in the lookup table. For example, a predetermined number of values ​​can be stored in the lookup table, and the value of the received input can be a value used as an address in the lookup table. For example, the address to be used can be determined from a predetermined number of MSBs of the input data.

[0104] The activation module is configured to calculate an output value based on the one or more entries read from the lookup table at step 404, and the output value represents the output of the activation function. The activation module can further calculate the output value based on at least a portion of the input data (e.g., a predetermined number of LSBs of the input data). The portion of the input data used to find one or more entries in the lookup table can be different from the portion of the input data used to calculate the output value. The activation module can calculate the output value by interpolating between two or more entries read from the lookup table.

[0105] The configuration data provided to the activation module may include one or more of: data points representing an activation function, data representing the shape of the function, or an identifier of a predetermined function to be implemented to allow the activation module to load the function from a database.

[0106] The hardware implementation 200 may also include a normalization module 265 and a pooling module 275, which are configured to share data via a shared buffer 270, which receives the output of the activation module 255. The normalization module may be configured to perform one or more normalization functions on the data in the shared buffer. The pooling module may be configured to perform one or more pooling functions on the data in the shared buffer - for example, a maximum or average calculation performed within a sliding window defined for each layer of the CNN.

[0107] The results stored in the shared buffer 270 are transferred to the memory interface 210, which may store the results in the external memory or transfer the results back to the input buffer for further processing without necessarily transferring them to the external memory first.

[0108] The shared buffer 270 allows these values ​​output by the activation module to be buffered until the correct samples are available to perform normalization and / or pooling. In this way, the shared buffer provides storage efficiency because in many implementations, normalization and pooling are not incorporated into the same process. In addition, the use of a shared buffer also allows the order of execution of normalization and pooling to be adapted to the CNN implemented at the hardware 200. In addition, by providing a shared buffer 270, activation, normalization, and pooling functions can be performed back-to-back without having to retrieve data from external memory.

[0109] Lookup Table

[0110] The configuration and use of the lookup table at the activation module 255 will now be described in detail. The teachings herein are applicable to any hardware that is arranged to perform an activation function in a DNN and uses a lookup table to represent the activation function performed.

[0111] The activation module 255 can be programmed with a specific function to be performed in a given hardware pass of an implementation of a CNN. The function to be implemented in the activation module (i.e., the activation function) can be viewed as a transfer function. The activation module 255 includes a lookup table that includes data representing the activation function. By modifying the data at the lookup table (e.g., according to configuration data provided to the activation module), the activation function performed by the activation module can be modified.

[0112] The activation function in DNN is generally nonlinear. Common activation functions used include the sigmoid function (e.g. Figure 5a Function shown), Figure 5b The tanh function shown, and Figure 4c The parameterized rectified linear function shown. In the case where the activation function is a parameterized rectified linear function, the activation module can be referred to as a parameterized rectified linear function (PReLU). PReLU is configured to generate an output element y as shown below i，j，k :

[0113] y i，j，k =f(x i，j，k ;w1,w2,b1,b2)=max{(w1*x i，j，k +b1),(w2*x i，j，k +b2)}

[0114] in, is a constant.

[0115] The activation module's lookup table is configured to store a data set representing an activation function, such that input values ​​to the activation module (or values ​​derived from those values) can be looked up in the table to identify the corresponding output value. Figure 5a and Figure 5b In , an input value x is used to look up in a table, and the output is the corresponding y value on the curve according to the representation of the curve stored in the lookup table. In some implementations, not all possible x and y values ​​are defined in the data stored at the lookup table, so an interpolation scheme is needed so that the output y can be determined from the data stored at the lookup table for any x value.

[0116] In the reference Figure 4d In the first approach to representing the activation function described above, the entries in the lookup table can be formed by at least one gradient and at least one offset. For example, the received data can represent a pair consisting of a gradient and an offset. In this way, the activation module 255 can implement the activation function at the lookup table as a set of pairs consisting of a value and a slope. Intermediate values ​​not stored in the lookup table can be derived using an interpolation method (e.g., linear interpolation).

[0117] The input value (e.g., from the accumulation buffer) can be used as an address in the lookup table. The lookup table can contain values ​​for samples C and gradients m corresponding to the points defined by those MSBs. Thus, the lookup table defines a plurality of data points on (or near) a curve representing an activation function, and a gradient representing the gradient of the curve at (or near) the defined data points. In order to minimize the overall error on a set of points derivable from a given pair of C and m values, it may be advantageous to store samples C that do not fall exactly on the curve and gradients m that do not accurately represent the slope of the curve at the defined data points at the lookup table. Appropriate C and m values ​​can be calculated for a given segment of the curve represented by the C and m values ​​to minimize the overall error on the segment. A segment can be a portion of the curve representing the activation function between two data points stored at the lookup table.

[0118] In one example, if the lookup table is a table of 64 or 65 entries, the 6 most significant bits (MSBs) of the input can be used as addresses in the lookup table. In another example, the lookup table can be a table of 256 or 257 entries, indexed by the 8 MSBs of the input. The lookup table can contain values ​​for sample C and gradient m at the points defined by these MSBs. The remaining least significant bits (LSBs) of the input can be interpreted as values ​​between 0 and 1, so that the output of the activation module is defined by the following equation:

[0119] y=m*(LSBs)+C

[0120] Figure 4dAn exemplary hardware implementation of the activation module 255 of the first approach is illustrated. The activation module 255 includes a lookup table configured to store values ​​representing the activation function implemented at the activation module 255. The activation module 255 also includes multiplication logic and addition logic to perform the calculations defined above. The MSBs of the input values ​​are used as addresses in the lookup table, and the corresponding m and C values ​​are read out and output from the lookup table. The remaining LSBs of the input values ​​are then multiplied with the lookup table output m, and the result is then added to the lookup table output C.

[0121] In the second approach, rather than storing m and C values ​​in a lookup table that are later retrieved, two different coarse output values ​​are determined that are located on either side of the input value. The output value to be output from the activation module is determined by interpolating between the two coarse output values ​​based on at least a portion of the input value, as described below with reference to Figure 4e and Figure 4f As described above. This approach may include storing a set of discrete output values ​​in a lookup table. Using the example described above of using the 6 MSBs of the input as addresses in the lookup table, there may be 65 different output values ​​in the lookup table for the activation function. Providing one more entry at the lookup table (e.g., providing 65 instead of 64 in the above example) than the number that can be addressed by the MSBs of the input values ​​used as indexes in the lookup table enables 64 spans between the entries to be defined. For example, the value in the 64th span may be determined by interpolating between the 64th and 65th entries. When an input value is received at the activation module, the six MSBs of the input value may be used as addresses to find the corresponding output value in the lookup table. In addition, the output value at the next address position in the lookup table may also be retrieved. In this way, the output values ​​of the data points stored in the lookup table adjacent to either side of the input point are identified.

[0122] like Figure 4f As shown, these MSBs of the input values ​​correspond to the input value x0, and the next corresponding input value is x1. Therefore, the two values ​​retrieved are the corresponding output values ​​for the identified input values, namely y0 and y1. Based on the retrieved output values ​​y0 and y1, these LSBs are used to identify the position of the input value x between x0 and x1 to allow interpolation between the output values ​​y0 and y1 to determine the corresponding output value y. Any suitable interpolation algorithm can be used.

[0123] Figure 4fAn exemplary hardware arrangement configured to perform a first interpolation approach is illustrated. In this arrangement, the activation module 255 includes a lookup table 410 configured to store output values ​​corresponding to a predetermined set of corresponding input values. The activation module 255 also includes an interpolation module 420 configured to perform an interpolation operation to calculate y. The interpolation may be a linear interpolation obtained according to the following formula:

[0124]

[0125] in:

[0126] x0 = MSBs of x

[0127] x1=MSBs of x+1

[0128] y0 = LUT(x0) (i.e., the output value retrieved from the lookup table for the input value x0)

[0129] y1=LUT(x1)

[0130] x-x0=LSBs of x

[0131] x1-x0 = known x spacing between data points in the lookup table

[0132] In some implementations, this spacing can be variable, thereby providing a larger number of data points in the case where the underlying activation function changes more quickly. If the spacing is fixed, the advantage is that the value of x1-x0 is represented by the value of 1 shifted left by the number of most significant bits used to look up in the lookup table. This allows the above equation to be implemented more simply in hardware as follows:

[0133] y=y0+(((y1-y0)*LSBs)>>number of MSBs)

[0134] like Figure 4f As shown, the activation module 255 is configured to receive an input value x. A predetermined number n of MSBs (corresponding to 2n entries in the lookup table) are used as addresses in the lookup table to retrieve the output value y0 in the corresponding address in the lookup table and the next output value y1 in the lookup table. The two values ​​y0 and y1 are output from the lookup table 410 and input to the interpolation module 420, which performs interpolation based on the remaining LSBs in the input value x to determine y. The interpolation module 420 then outputs an output value y corresponding to the input value x.

[0135] Figure 4g Another hardware implementation of the activation module 255 is shown for performing quadratic interpolation between data points stored in the lookup table. Figure 4gIn the arrangement of , the lookup table is configured to retrieve three output values, namely c0, c1 and c2, based on those MSBs of the input value x. This enables, for example, quadratic interpolation to be performed by the interpolation module 420. In this way, more complex curves can be interpolated by retrieving more coefficients from the lookup table 410. It will be appreciated that more than three values ​​can be retrieved and used to interpolate output values ​​for input values.

[0136] Other interpolation schemes may also be used.

[0137] Through Figure 4f and Figure 4g Different data points written to the lookup table 410 shown may represent different activation functions in the lookup table, and thus different activation functions may be executed by the activation module. Thus, the use of a lookup table enables the specific activation function to be executed by the activation module 255 to be configured at runtime. This is not the case for fixed function hardware configured to perform an activation function on data entering the activation module, where the use of a lookup table consumes much less chip area and power than programmable logic configured to implement the activation function. Typically, for a basic activation function represented at the lookup table, the accuracy of the output of the activation module increases with the number of data points stored at the lookup table. However, storing a larger number of data points increases the size of the lookup table. For a given hardware implementation, a reasonable compromise can be established between accuracy and area consumption. Figure 2 In the exemplary hardware shown for implementing a CNN, it has been found that a lookup table configured to store approximately 256 data points provides a good balance of characteristics in a system where the activation layer operates on 16-bit data.

[0138] Figure 3 An exemplary architecture for the lookup table 301 is shown. The architecture shown is particularly suitable for Figure 4f , wherein two adjacent entries are looked up in the lookup table. Lookup table 301 includes two databases 307 and 308. These databases are single-port synchronous random access memories (RAMs). Database 307 is arranged to store even-numbered entries, which represent a set of data points that define an activation function; database 308 is arranged to store odd-numbered entries, which represent a set of data points that define an activation function. For example, when numbering from 0 (e.g., from 0 to 64 to provide 65 entries), for a lookup address of length N bits, the odd-numbered database may include odd-numbered entries, the even-numbered database can include The arbitrator 306 is configured to arbitrate between the read 302 and write 304 addresses entering the lookup table.

[0139] Read address 302 represents the address of the entry in the lookup table, which can include a predetermined number of most significant bits of the input value x. In other implementations, the read address can be derived from the input value x in any other way at arbiter 306 or another unit. The larger the number of MSBs used, the larger the address space, and the larger the number of possible entries in the lookup table. For how many MSBs are used as indexes and how many LSBs are used for interpolation, a choice can be made to achieve a compromise between precision and the size of the lookup table. For example, for an activation module operating with 16 bits, using the first 6 MSBs as the lookup table address can be used to provide approximately 64 entries in each of databases 307 and 308. The remaining 10 LSBs can be used to interpolate between the outputs provided by the lookup table. In some implementations, translating the input data into the address in the lookup table can be performed at a unit outside the arbiter.

[0140] The write address 304 may include the address of the entry to be updated in the lookup table, and the data value to be written to the databases 307 and 308 is provided by the lookup table input 305 .

[0141] The lookup tables 301 may be provided at an activation module in hardware configured to process layers of a deep neural network in multiple passes. Depending on the capabilities of the hardware, each pass may execute one or more different layers (e.g., convolutional layers, activation layers, normalization layers, etc.). At each pass through an activation layer, one or more activation modules tasked with processing that layer are configured to execute the activation functions loaded into the lookup tables of these activation modules. These lookup tables may be updated before each pass through the activation layer so that a different set of activation functions is executed at each pass. In order to update the lookup tables while avoiding delaying processing, it is advantageous to write values ​​for the next pass to the lookup tables while the values ​​for the current pass are being used. This may be achieved, for example, by providing multiple lookup tables at the activation module or by providing multiple data sets at the lookup tables of the activation modules.

[0142] exist Figure 3In the example shown, databases 307 and 308 each contain two complete data sets, so that when one of these data sets is being used in a pass of the activation layer, the other can be updated. Index 303 is used to indicate to the arbitrator: in a given pass, which of the two sets of entries at databases 307 and 308 is to be used. The set of database entries that is not being used in the current hardware pass can be written. The index can be updated in any suitable manner-for example, at the lookup table or in the activation module itself. As each pass of the activation layer is executed, the index can be updated to alternate between the two data sets at the lookup table of each activation module. It may not be necessary to load a new table at each hardware pass-for example, if the activation function is the same, a new lookup table does not need to be loaded. The index can be configurable for each pass to indicate which of the two entry sets at the lookup table should be applied. When a new activation function is required for the next hardware pass, a new set of entries may be written to a database that is not currently in use, and the index updated when the next hardware pass is configured so that the lookup table uses the database that has had the new entries written to it. Writing new entries to the lookup table and the indexing for each pass is performed according to the definition of the neural network being executed, and may be implemented at the hardware 200 by providing an appropriate command stream to the command decoder 220.

[0143] The arbiter can be configured to give higher priority to read addresses 302 than write addresses 304 to avoid delays introduced to the operation of the neural network while new data is being written to the lookup table. This enables writes to be performed in the gaps between reads being performed related to neural network processing.

[0144] The write data 305 provided to the lookup table may include values ​​in a form suitable for being written directly into entries in the lookup table. The activation module 255 may be configured to receive the lookup value from the command decoder 220, which is arranged to receive the value of the function from the external memory via the memory interface 210. The activation module 255 may be configured to receive a different set of values ​​to be stored in the activation layer for each hardware pass. In some configurations, the DNN may include multiple activation layers. In the hardware 200 implementing the CNN, the command decoder 220 may be configured to receive values ​​that define the functions to be used for one or more activation layers of the CNN in a hardware pass.

[0145] The read address 302 may be interpreted by the arbiter 306 (or another unit) as a pair of addresses: the first address represents a search for an even entry of the database 307, and the second address represents a search for an odd entry of the database 308. It may be advantageous to configure the databases such that when the read address read_addr is even, the two databases read from the same address, and when the read address read_addr is odd, one database reads from the address of read_addr+1 (or -1), or vice versa. In other implementations, a pair of addresses may be used where the number of bits that differ in the pair is not 1. Whether the read address is odd or even is identified by the least significant bit of the address: for example, if read_addr[0]=0, it is even, otherwise it is odd.

[0146] The outputs from these databases can be provided to multiplexers 312 and 313, which select the appropriate outputs y0 and y1 depending on whether the read address is even or odd. For example, in the above scenario, if the read address is even, the output of the even database 307 is used as y1 and the output of the odd database 308 is used as y0, otherwise, the output of the even database 307 is used as y0 and the output of the odd database 308 is used as y1. Signal 314 indicates the parity of the read address and provides a control signal to multiplexers 312 and 313 - for example, it can be the least significant bit of the read address, read_addr[0]. This approach ensures that the output y0 is always less than the output y1, thereby simplifying the interpolation between these values, because the above formula can be directly applied to these outputs. In other implementations, the output values ​​y0 and y1 can be in any particular order.

[0147] Signal 314 may be pipelined to align it with the output read from databases 307 and 308 .

[0148] To avoid stalling problems, FIFO buffers (FIFO 310, 311) may be provided at the output of these databases, and a FIFO buffer (FIFO 309) may be provided in parallel with these databases on the data path of signal 314. FIFO 309 may be configured to match the pipeline length of the address to the path through RAM 307 and 308, and FIFO 310 and 311 may be configured to handle a fixed 2 cycle delay at RAM 307 and 308. This enables the lookup table to be stalled back like a normal pipeline. Each of these FIFOs may have a depth of 2 entries.

[0149] It will be appreciated that the above approach may be extended to activation modules including more than two lookup tables (eg, where it is advantageous to read more than two values ​​simultaneously—eg, if the activation function were a surface of more than two dimensions).

[0150] The inputs to the activation layers in a DNN can in principle take any value within the range allowed by the bit length of these inputs. However, in practice, for a specific implementation of the DNN, the input values ​​to the activation layers may only take a subset of the possible range of input values. This causes some parts of the lookup table to be unused, which is an inefficient use of chip area. By arranging so that the entire range of entries in the lookup table can be used, the effective accuracy of the activation function performed with the lookup table can be improved, and the inefficient use of storage space in the lookup table can be largely eliminated.

[0151] During the training of a DNN, the weights, activation functions, and other parameters of the system (e.g., bias, minimum / maximum clamp values) are determined for those layers of the neural network. This is usually done by programming the system in a data processing system (e.g., Figure 6 The system includes hardware configured to implement the DNN (e.g., Figure 2 Hardware 200). In some cases, the hardware can be simulated in software, and training software is arranged to operate on the hardware simulation to establish configuration parameters to be used at the hardware itself. Generally speaking, the training software will provide a test data set to the DNN and iteratively modify the parameters of the DNN to optimize its performance. For example, in the case where the DNN is configured to perform object recognition in an image, a set of activation functions and an initial layer configuration of the DNN can be determined by a user. The training software can then be used to provide the DNN with a data set consisting of thousands of different images of the object (e.g., a cup), while introducing small changes to the parameters of the DNN according to some predetermined scheme. Generally speaking, those changes that improve the performance of the DNN (e.g., the accuracy with which it recognizes a cup in an image) are retained, while those changes that deteriorate the performance of the DNN are discarded. In this way, an optimized set of parameters can be defined for the DNN.

[0152] Additionally, it is advantageous to determine the expected range of input values ​​to the activation layer of the optimized DNN - that is, the range of input values ​​observed to be provided to the activation module, rather than the possible range of input values ​​allowed by the bit length of the input values ​​to the activation module. This can be achieved by performing a range calibration at a configuration module. Once the DNN has been optimized, a range calibration can be performed. Range calibration can be performed by a configuration module that causes calibration data to be provided to the activation module (e.g., by providing test data to the DNN for processing) and monitors the input values ​​provided to the activation module of the DNN during operation of the DNN to identify the range of input values.

[0153] Range calibration can be performed by a hardware expression. The expression of hardware 200 can be any combination of software and / or hardware (e.g., a programmable chip). For example, hardware 200 can be simulated by software at a data processing system, calibration data is provided to such hardware simulation, and the range of input values ​​to the simulated activation module is monitored to identify the range of input values. This hardware simulation can be the same hardware simulation used to train the DNN. Range calibration can be performed as part of DNN training—for example, as a training phase after optimization of filter weights and other parameters. Simulating hardware for implementing DNNs can include simulating the hardware configured to execute the DNN, or simulating the DNN to be implemented in hardware. A configuration module configured to provide calibration data and monitor the resulting input values ​​can be provided in software—for example, in training software for optimizing DNNs.

[0154] In other examples, a data processing system supporting training software may include hardware 200, and the training software may be configured to perform range calibration using hardware 200 configured to execute a DNN. The DNN implemented at the hardware may be optimized after training performed by the software. The range of input values ​​to the activation modules of the hardware when executing different activation functions of the DNN on calibration data may be captured by the training software to identify the input range for each activation module when executing each training function.

[0155] To provide access to input values ​​to the activation module, the hardware implementing the DNN (or a representation thereof - e.g., a software emulation of the hardware) can be configured to copy the input data to the activation module onto a data path consumed by an entity configuration module (e.g., the training software) that monitors the input values ​​and forms an expected input range for the activation module. For example, a copy of the input data to the activation module can be streamed to a memory accessible to the monitoring entity. Figure 2In the specific example shown, the output from the accumulation buffer 250 (which represents the input to the accumulation module 255) can be directed through the data path 282. Using the memory interface 210, the data path 282 can direct the input data to the data processing system including the hardware 200 (see Figure 6 600) to be consumed by monitoring software running at the data processing system. Once the configuration of the DNN is complete, the data path 282 can be turned off. The data path 282 can be used only during the range calibration or other training phase of the DNN.

[0156] In other examples, a configuration module is provided to determine the range of input values ​​to the activation module during its operation, and the configuration module may be fixed or programmable hardware.

[0157] One approach to determining the range of input values ​​at the activation layer is to capture the maximum and minimum input values ​​in the stream of input values ​​provided to the activation module. Then, the range of the determined input values ​​can be set to be equal to or approximately equal to the range of values ​​between these maximum and minimum values ​​captured for the activation module. In other examples, the stream of input values ​​can be processed to form a probability distribution or histogram, which represents the possibility of each input value occurring. For many applications, this probability distribution may be generally a normal distribution, or (if the probability distribution is not symmetrical) an asymmetric normal distribution. The range of the determined input values ​​can be set for the activation module based on this probability distribution. For example, according to the probability distribution, the determined range can be set to cover 90%, 95%, 98%, 99% or 99.5% of the input value range, wherein the minimum and maximum values ​​are symmetrical about the center of this distribution or are input values ​​corresponding to equiprobable points on the probability distribution curve. In another example, the determined range can be set to a range of input values ​​where the maximum and minimum endpoints of the range are defined as input values ​​at or near where the probability distribution falls below a predetermined or programmable level (e.g., 1%, 0.5%, 0.25%, 0.15%, 0.05%, or 0.005%). It will be appreciated that many variations are possible. The configuration module can be configured to determine a range of input values.

[0158] Since range calibration can be performed as a further training phase prior to use of the DNN, it is generally not important whether monitoring the input values ​​to the activation module introduces latency into the system (e.g., due to the data stream 282 provided back to the memory via the memory interface 210). In addition, if appropriate, the range calibration can be performed in the data processing system (e.g., Figure 6 600 ) performs very time-consuming and potentially time-consuming processing on the input data to identify the optimal range of values ​​for the activation module of the hardware (e.g., hardware 200) used to implement the DNN.

[0159] Figure 6 Shows Figure 2 602, a memory 609 may be provided with the processor 608 for storing training data, input data, configuration parameters for the hardware 200, etc. for processing at the trained CNN implemented at the hardware 200. A bus or other interconnect 603 may be provided to connect the processor, memory, and hardware 200. The figure shows a flow 282 of input data for one or more activation modules to a memory for storage for processing by the training software 604, a flow 607 of input data and returned processed data, and a flow 606 of configuration parameters to the hardware 200.

[0160] Once the input range is determined for the activation module, the calibration module (e.g., training software 604) is configured to calculate appropriate lookup data (e.g., a set of data points) to store at the lookup table of the activation module. For example, the set of data points defining the activation module can be calculated to be evenly spaced over the expected input range of input to the activation module configured to perform the activation function, or the data points can be calculated to be within the determined input range and the data points are more closely spaced in regions where the activation function changes more rapidly than in regions where the activation function changes less rapidly.

[0161] Once determined, the lookup data may be written to a repository of configuration data (e.g., in memory 602) for use at the activation module during execution of the DNN when the corresponding activation function is to be executed at the activation module. For example, during execution of the DNN, when the DNN requires execution of an activation function at the activation module in question, the lookup data is written to the lookup table of the activation module so that the activation module can perform an approximation of the activation function over the determined input range.

[0162] This range calibration may be repeated across all activation functions used in the DNN to establish optimal lookup data for representing each activation function. Furthermore, the same activation function may be used at the same activation module (but in different hardware passes executed at different stages of the DNN processing model) when processing a set of input data at the DNN. The input range of data to an activation module executing the same activation function (but at different points in the processing of the DNN) may be different between these processing points. Therefore, during processing of the DNN, it may be advantageous to perform a range calibration for each (or at least some) of the various examples of activation functions executed at the activation module. In this case, multiple sets of lookup data may be determined for each activation function to be executed at the activation module.

[0163] The processing and range calibration of the hardware implementation 200 will often be performed at a different system than the system that ultimately uses the trained, calibrated DNN. For example, a given hardware implementation of a DNN may be trained with a first system (e.g., using a representation of the hardware, which may be the hardware itself, or a simulation in software, or programmable hardware), and lookup data may be generated for the activation module of the hardware over a determined input range according to the principles described above. This lookup data (configured to best represent the activation function at the lookup table of this activation module) and the definition of the DNN (e.g., with its weights and other parameters) may then be provided for use at a second system that includes the hardware implementation of the DNN. For example, the first system may be a training system operated by a manufacturer of a data processing system (e.g., a smartphone) that includes the hardware 200, and the second system may be the data processing system (e.g., a smartphone) itself. In this way, the hardware implementation of the second system is optimally configured to perform the activation function at its activation module, maximizing the performance of the hardware.

[0164] Because the input range determined for the activation module is determined in dependence on training data or other data that represents data that the DNN will process at runtime, the following situation is possible (if the determined input range is not the complete possible range given the bit length of the input value): During use, the activation module may actually receive input values ​​that are outside the determined range. For many functions, values ​​of the activation function that are outside the determined range can be approximated by function values ​​at the boundaries of the range. In other words, for input values ​​outside the determined range, the activation function can be approximated outside the determined range by clamping the corresponding output value to the activation function value at the boundary of the determined range.

[0165] For example, Figure 7The tanh activation function is illustrated, for which the range of input values ​​has been determined according to the above approach as a range 705 between boundaries 701 and 702. Boundaries 701 and 702 may or may not be symmetrical about the midpoint x=0 of the tanh function. For input values ​​outside the determined range, the activation function is clamped at clamped outputs 703 and 704, which are output values ​​at the boundaries of the determined range 705. Clamping for input values ​​outside the determined range can be performed in the following manner: a lookup table is configured to return output values ​​stored for input values ​​at the boundaries of the determined input range for input values ​​that fall outside the determined input range. In some examples, clamping for input values ​​outside the determined range can be performed at other logic and bypass the lookup table (e.g., in Figure 2 In the example shown, at simple path 281 described below). For example, a clamp circuit can be provided to clamp input values ​​that fall outside the determined range to the closest boundary input value of the determined input value range, so that the lookup table returns the activation function value at this closest boundary input value for those values.

[0166] In some examples, exception logic can be provided (e.g., in logic associated with the lookup table in the activation module) to determine when an input value falls outside of a determined range, and in response to this determination, cause the activation module to output a clamped value representing the value of the activation function at the boundary of the determined range. The clamped value can also be programmed into the activation module along with the lookup data of the lookup table.

[0167] In some examples (e.g. Figure 7 In the example shown), a better approximation can be achieved for the activation function outside the determined input range by using a straight line with a non-zero gradient, for example by extrapolating the gradient of the function at the boundaries of the determined range, or by identifying the gradient that minimizes the error between the boundaries of the determined range and the corresponding maximum / minimum possible input value. Figure 7 In the region of input values ​​1 to 2 and -1 to -2, the activation function is more closely approximated by lines 706 and 707. At the point where line 706 reaches the maximum value of the activation function (at Figure 7 In the example shown, at a point outside of 1), this approximation can be clamped to this maximum value. The same is true for the minimum value of the activation function and the straight line 707. Alternatively, the gradients of the straight lines 706, 707 that approximate the activation function outside the determined range 705 can be selected so that: at the maximum possible input value according to the bit length of the input number, the approximation of the activation function takes the maximum possible value of the activation function (e.g. 1 for tanh). The same is true for the minimum possible value of the activation function and the straight line 707.

[0168] Straight line approximation for functions outside of the determined range may be implemented at logic including multipliers and adders. The gradient and offset for extrapolating the activation function may be determined during training—e.g., at training software 604 when determining the range of input values ​​for the activation module. Configuration parameters may be generated during training for writing to the activation module to configure its logic to perform straight line approximation based on the determined gradient and offset.

[0169] An activation layer may include multiple activation modules running in parallel. It may be advantageous to independently determine the expected input value range for each of these activation modules and thus potentially provide different lookup data to the lookup tables of different activation modules performing the same activation function in the same layer. Alternatively, the input ranges determined for each of these activation modules may be combined to form one expected input range for the activation layer as a whole, which applies to all activation modules of the activation layer.

[0170] The input values ​​operated by the activation layer of the DNN are usually different for different applications. For example, in one application, the activation layer may receive values ​​strictly in the range of 0 to 1, while in another application, the activation layer may receive values ​​anywhere in the range of 192 to 256. In order to make the size of the input values ​​stored in the lookup table as small as possible and simplify its implementation, it may be advantageous to subtract a programmable offset from the input values ​​to the activation module before looking up the input values ​​to the activation module in the lookup table. In this way, the input values ​​used as indexes to the lookup table can be kept small. For example, in the case where the data values ​​received by the activation module are in the range of 192 to 256, the activation module can be programmed to subtract 224 from each incoming data value to form input values ​​arranged around 0 for its lookup table. The appropriate offset can be determined during training for a given activation module (e.g., by the software 604 when the input value range to the activation module is determined by the software 604), and the activation module is configured with the offset at runtime (e.g., it can be provided with other configuration parameters, such as maximum / minimum clamp values ​​and / or lookup data for the lookup table). Adder logic may be provided at the activation module to perform a subtraction on the determined offset.

[0171] It will be appreciated that there are other approaches to implementing activation functions using lookup tables that would also benefit from the range calibration described herein.

[0172] In some implementations, the lookup table and its associated logic can be viewed as providing a "complex path" 280 through the activation module. The activation module 255 can also include a simple path 281 that provides an alternative mechanism for performing the activation function. For example, the simple path can implement a rectified linear unit (ReLU) that is configured to implement a ReLU function. In the ReLU function, by as shown below and as Figure 4b As shown, the maximum value is identified to calculate the output element y i，j，k :

[0173] y i，j，k =f(x i，j，k )=max{0,x i，j，k}

[0174] Therefore, for values ​​of x less than 0, y = 0. The ReLU function can be considered as clamped for negative input x. The ReLU function can be implemented in a conventional manner using a combination of multipliers and adders.

[0175] When the hardware pass of the CNN to be implemented uses a ReLU function implemented on a simple path 281, rather than a lookup table 410 and its associated logic implemented on a complex path 280, it may be advantageous to bypass the complex path and shut down (e.g., stop providing a clock to) the lookup table and its associated logic to reduce power consumption. Similarly (although the benefit is relatively small), when an activation function is performed using a lookup table, it may be advantageous to shut down the computational unit. However, when an activation function is performed using a lookup table for a complex path, it may be advantageous to clamp the input value to a minimum and / or maximum output value with the simple path outside a predetermined or programmed range of input values ​​to the activation module (e.g., outside the range in which the lookup table operates) as described above. Therefore, it may be advantageous to arrange the simple path and the complex path so that data passes through the simple path before the complex path (if not bypassed).

[0176] Figure 8 The flowchart of the method described in this article shows the method described in this article. Figure 6 The data processing system shown is executed at Figure 2 Preferably, the conventional optimization of the neural network provided at the hardware implementation is performed after Figure 8 method, so that the parameters that will be used during the operation of the DNN have been established.

[0177] Calibration data (e.g., test data representing a typical range of inputs to the DNN during operation of the DNN) is provided to a representation of hardware for implementing a DNN (801). The hardware includes activation modules that use lookup tables to approximate activation functions to be performed. By monitoring input values ​​to the activation modules of the hardware in the representation (802), a range of input values ​​to the activation modules can be determined (e.g., according to any of the above approaches). Lookup data is then generated for the lookup table (803), which represents the activation function over the determined range of input values. This enables the number of entries in the lookup table to be fully utilized over the possible range of input values ​​to the activation module. The range calibration of steps 801-803 can be performed at a configuration module (e.g., software running at a training system) that is suitable for use with a representation of the hardware for implementing the DNN. These range calibration steps are typically performed during a calibration phase during training of the DNN.

[0178] During operation of the DNN (e.g., when the optimized DNN is implemented at hardware 200 according to established filter weights and other configuration parameters (including the lookup data required to represent the desired activation function at the lookup table of the activation module) and as it receives an input data stream (805), the lookup data generated for the activation function at the activation module is loaded (804) into the lookup table on the activation module being requested to execute the activation function. This enables the lookup table to execute the activation function over the determined input range (806) without wasting lookup table entries on input values ​​that are unlikely to be received at the activation module.

[0179] This method may be performed for all activation modules of a hardware implementation, and / or may be performed for each activation function required by the DNN that is executed at each activation module.

[0180] In some embodiments, the methods described herein and / or the data processing systems described herein may be implemented entirely in hardware—eg, in the form of integrated circuits.

[0181] Figure 2 , 3 , 4d, 4f and 4g are illustrated as including a plurality of logic blocks. This is merely illustrative and is not intended to define a strict division between the different logic elements of these entities. Each functional block may be provided in any suitable manner. It should be understood that the intermediate values ​​described herein as being formed by the hardware logic need not necessarily be actually generated at any point, but may simply represent logical values ​​that facilitate the description of the processing performed by the logic between its inputs and outputs.

[0182] The hardware logic described herein can be implemented in hardware on an integrated circuit. A data processing system including the hardware logic described herein can be configured to perform any of the methods described herein. In general, the functions, methods, techniques or components used to train and implement the above-mentioned deep neural network in a data processing system can be implemented in software, firmware, hardware (e.g., fixed logic circuits) or any combination thereof. The terms "module", "functionality", "component", "element", "unit", "block" and "logic" may be used herein to refer to software, firmware, hardware or any combination thereof. Those parts described herein that can be implemented in software can be program codes that perform the specified tasks when executed on a processor. The algorithms and methods described herein can be performed by one or more processors that execute code, which causes this (these) processors to perform these algorithms / methods. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices, which may use magnetic, optical, and other technologies to store instructions or other data that can be accessed by a machine.

[0183] As used herein, the terms "computer program code" and "computer-readable instructions" refer to any type of executable code for a processor, including code expressed in a machine language, an interpreted language, or a scripting language. Executable code includes binary code, machine code, bytecode, code that defines an integrated circuit (such as a hardware description language or a netlist), and code expressed in a programming language code (such as C, Java, or OpenCL). Executable code can be, for example, any type of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, executed at a virtual machine or other software environment, causes a processor of a computer system that supports the executable code to perform the tasks specified by the code.

[0184] A processor, computer, or computer system may be any kind of device, machine, or special purpose circuit, or a collection or part thereof, and has processing capabilities that enable it to execute instructions. A processor may be any kind of general purpose processor or special purpose processor, such as a CPU, GPU, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0185] It should also be considered to cover software (e.g., HDL (Hardware Description Language) software) that defines the hardware configuration described herein, which is used to design integrated circuits or configure programmable chips to perform the desired functions. That is, a computer-readable storage medium can be provided on which computer-readable program code is encoded in the form of an integrated circuit definition data set, which code, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware logic that is configured to perform those parts of any method described herein that are described as being performed in hardware, or to manufacture the hardware logic of any apparatus described herein. The integrated circuit definition data set can be, for example, an integrated circuit description.

[0186] A method of manufacturing the hardware logic described herein at an integrated circuit manufacturing system may be provided. An integrated circuit definition data set may be provided that, when processed in an integrated circuit manufacturing system, causes execution of such a method of manufacturing hardware logic.

[0187] The integrated circuit definition data set may be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language that defines an integrated circuit at any level, including register translation level (RTL) code, as a high-level circuit representation form (such as Verilog or VHDL), and as a low-level circuit representation form (such as OASIS (RTM) and GDSII). The higher-level representation forms logically define the integrated circuit (such as RTL) and can be processed at a computer system that is configured to generate a manufacturing definition of the integrated circuit in the context of a software environment, including definitions of circuit elements and rules for combining these elements to generate a manufacturing definition of the integrated circuit defined by the representation form. The typical situation is that software is executed at a computer system to define a machine, and one or more intermediate user steps (such as providing commands, variables, etc.) may be required to cause the computer system configured to generate the manufacturing definition of the integrated circuit to execute the code that defines the integrated circuit to generate the manufacturing definition of the integrated circuit.

[0188] Now refer to Fig. 9 An example of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system to manufacture hardware logic is described below.

[0189] Fig. 9An example of an integrated circuit (IC) manufacturing system 1002 is shown, which is configured to manufacture the hardware logic described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition data set (e.g., defining the hardware logic described in any of the examples herein), process the IC definition data set, and generate an IC (e.g., implementing the hardware logic described in any of the examples herein) based on the IC definition data set. The processing of the IC definition data set configures the IC manufacturing system 1002 to manufacture an integrated circuit that implements the hardware logic described in any of the examples herein.

[0190] The layout processing system 1004 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout from an IC definition data set are known in the art and may include, for example, synthesizing the RTL code to determine a gate-level representation of the circuit to be generated, for example in the form of logic components (e.g., NAND, NOR, AND, OR, MUX, and flip-flop components). The circuit layout may be determined from the gate-level representation of the circuit by determining location information for these logic components. This may be done automatically or with user intervention to optimize the circuit layout. When the layout processing system 1004 determines the circuit layout, it may output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0191] IC generation system 1006 generates an IC based on a circuit layout definition, as is known in the art. For example, IC generation system 1006 may implement a semiconductor device manufacturing process to generate an IC, which may include a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually created on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that may be used in a photolithography process to generate an IC based on the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of a computer-readable code that the IC generation system 1006 may use to form a suitable mask for use in generating an IC.

[0192] The different processes performed by the IC manufacturing system 1002 may all be implemented in one location, such as by one party. Alternatively, the IC manufacturing system 1002 may be a distributed system such that some of these processes may be performed at different locations and by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask according to the circuit layout; and (iv) manufacturing the integrated circuit using the mask.

[0193] In other examples, processing of the integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture YYY without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor (e.g., an FPGA), and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0194] In some embodiments, the integrated circuit manufacturing definition data set, when processed in the integrated circuit manufacturing system, can cause the integrated circuit manufacturing system to generate the device described herein. Fig. 9 Configuring an integrated circuit manufacturing system in the manner described may enable the devices described herein to be manufactured.

[0195] In some examples, the integrated circuit definition data set may include software that runs on or in conjunction with hardware defined at the data set. Fig. 9 In the example shown, the IC generation system can also be configured by the integrated circuit definition data set to load firmware onto the integrated circuit according to the program code defined in the integrated circuit data set when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use in the integrated circuit.

[0196] Compared to known implementations, implementing the concepts set forth in this application in devices, equipment, modules and / or systems (and in methods implemented herein) can result in performance improvements. These performance improvements can include one or more of the following: improving computing performance, reducing latency, increasing yield, and / or reducing power consumption. During the manufacture of these devices, devices, modules and systems (e.g., in integrated circuits), performance improvements and actual implementations can be compromised to improve manufacturing methods. For example, performance improvements can be compromised with layout area to be comparable to known implementations, but using less silicon. This can be achieved, for example, by reusing functional blocks in a serialized manner or sharing functional blocks between elements of devices, devices, modules and / or systems. Conversely, the concepts set forth in this application that result in improvements in actual implementations of devices, devices, modules and systems (e.g., reducing silicon area) can also be compromised with improved performance. This can be achieved, for example, by manufacturing multiple instances of a module within a predetermined area budget.

[0197] The applicant discloses each individual feature described herein and a combination of two or more of these features (as long as these features or combinations can be implemented based on the entire specification and the common knowledge of those skilled in the art, regardless of whether these features or feature combinations can solve any problem disclosed herein). According to the above description, it is obvious that those skilled in the art can make various changes within the scope of the present invention.

Claims

1. A method of configuring hardware for implementing a deep neural network (DNN) to perform an activation function, the hardware comprising, at an activation module for performing the activation function, a programmable lookup table for storing lookup data that approximates the activation function over a first range of input values ​​to the activation module, the method comprising: providing calibration data to a representation of the hardware and operating the DNN on the calibration data; During operation of the DNN on the calibration data, monitoring input values ​​provided to an activation module of the hardware representation to determine a range of input values ​​to the activation module; generating lookup data for the lookup table, the lookup data representing the activation function over the determined range of input values; as well as The generated lookup data is loaded into a lookup table of the hardware, thereby configuring an activation module of the hardware to perform the activation function over the determined range of input values.

2. The method of claim 1, further comprising: receiving an input data stream to the hardware; as well as The activation function is performed over the determined range of input values ​​using the lookup table.

3. The method of claim 2, wherein: The DNN uses the activation module to execute multiple activation functions to process the input data stream, and the method also includes: repeating the providing, monitoring, generating and loading steps for each activation function to generate and load lookup data for a lookup table, wherein the lookup data represents each of the multiple activation functions.

4. The method according to claim 2 or 3, wherein: The lookup table includes two sets of lookup data and is capable of switching between the two sets of lookup data. When the activation module executes a series of activation functions, the first activation function in the series and the loading of the generated lookup data for the next activation function in the series into the lookup table are executed simultaneously.

5. The method according to claim 2 or 3, further comprising: Upon receiving an input data stream to the hardware, a check is made as to whether the input value to the activation module falls outside a determined range of input values, and if the input value to the activation module falls outside a determined range of input values, an activation function value corresponding to the closest extreme value of the determined range of input values ​​is used as an output value of the activation function.

6. The method according to claim 2 or 3, wherein: Monitoring the input to the activation module also includes: determining an offset, which, when subtracted from each input value to the activation module, causes the range of the input values ​​to be substantially centered around a predetermined input value, and executing the activation function also includes: subtracting the offset from each input value received at the activation module before looking up each input value in the lookup table.

7. The method according to claim 2 or 3, wherein: Executing the activation function includes: upon receiving a first input value, searching the lookup table for a pair of adjacent data points closest to the first input value, and interpolating between a pair of corresponding values ​​of the activation function to form a value estimate of the activation function corresponding to the first input value.

8. The method of claim 7, wherein: A predetermined number of most significant bits of the first input value are used as lookup addresses in the lookup table and remaining bits of the first input value are used in interpolation between the pair of corresponding values ​​of the activation function.

9. The method of claim 7, wherein: The lookup table includes a first database and a second database, the first database includes a first set of data points, the second database includes a second set of data points, so that for each adjacent pair of data points, one of the data points is in the first database and the other data point is in the second database, and executing the activation function for the first input value includes: simultaneously searching for each of the pair of adjacent points in the first or second database of the pair of adjacent points.

10. The method according to claim 1 or 2, wherein: Generating the lookup data includes computing a set of curves that approximate the activation function over the determined range of input values, each curve representing a portion of the activation function, such that the set of curves together identify an output value for each input value within the determined range.

11. The method of claim 10 as dependent on claim 2, further comprising: Upon receiving an input data stream to the hardware, checking whether the input value to the activation module falls outside the determined range of input values, and if the input value to the activation module falls outside the determined range of input values, extrapolating the closest curve in the set of curves to provide an output value of the activation function.

12. The method according to claim 1 or 2, wherein: Monitoring inputs to the activation module includes identifying maximum and minimum input values ​​to the activation module and using the maximum and minimum input values ​​as boundaries for a determined range of input values.

13. The method according to claim 1 or 2, wherein: Monitoring inputs to the activation module includes forming a histogram of input values ​​representing the probability of occurrence of input values ​​and using a pair of input values ​​between which a predetermined or programmable portion of the input value distribution falls as boundaries of a determined range of input values.

14. The method according to claim 1 or 2, wherein: The determined range of input values ​​is smaller than a range of possible input values ​​based on a bit length of the input values, and the lookup data represents the activation function over a range smaller than the possible input values.

15. The method according to claim 1 or 2, wherein: The method is performed following optimization of the DNN.

16. A data processing system comprising: A hardware representation for implementing a deep neural network (DNN), the hardware representation comprising an activation module for executing an activation function, the activation module comprising a programmable lookup table for storing lookup data, the lookup data representing the activation function; as well as Configuration modules are configured to: causing calibration data to be provided to a representation of the hardware and operating the DNN on the calibration data; During operation of the DNN on the calibration data, monitoring input values ​​provided to an activation module of the representation to determine a range of input values ​​to the activation module; as well as generating lookup data for the lookup table, the lookup data representing the activation function over the determined range of input values; Wherein, when loaded into a lookup table of an activation module of the hardware, the generated lookup data is suitable for causing the activation module to perform the activation function over a determined range of input values.

17. Hardware for implementing a deep neural network (DNN), comprising an activation module for executing an activation function, the activation module having a programmable lookup table for storing lookup data representing the activation function, wherein in use, the activation module is configured to load first lookup data into the lookup table, the first lookup data being generated over a determined range of input values ​​to the activation module for executing the activation function, wherein The range of input values ​​to the activation module is determined by monitoring input to the activation module of the hardware representation, the activation module being arranged to operate on calibration data.

18. A method of using an integrated circuit manufacturing system to manufacture hardware for implementing a deep neural network (DNN), the hardware comprising an activation module for executing an activation function, the activation module having a programmable lookup table for storing lookup data representing the activation function, wherein in use, the activation module is configured to load first lookup data into the lookup table, the first lookup data being generated over a determined range of input values ​​to the activation module for executing the activation function, wherein The range of input values ​​to the activation module is determined by monitoring input to the activation module of the hardware representation, the activation module being arranged to operate on calibration data, the method comprising: processing the computer-readable description of the hardware using a layout processing system to generate a circuit layout description of an integrated circuit implementing the hardware; and The hardware is manufactured according to the circuit layout description using an integrated circuit generation system.

19. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which when executed at a computer system causes the computer system to perform a method of configuring hardware for implementing a deep neural network (DNN) to perform an activation function, the hardware comprising a programmable lookup table at an activation module for performing the activation function for storing lookup data, the lookup data approximating the activation function over a first range of input values ​​to the activation module, the method comprising: providing calibration data to a representation of the hardware and operating the DNN on the calibration data; During operation of the DNN on the calibration data, monitoring input values ​​provided to an activation module of the hardware representation to determine a range of input values ​​to the activation module; generating lookup data for the lookup table, the lookup data representing the activation function over the determined range of input values; as well as The generated lookup data is loaded into a lookup table of the hardware, thereby configuring an activation module of the hardware to perform the activation function over the determined range of input values.

20. An integrated circuit manufacturing system, comprising: a non-transitory computer readable storage medium having stored thereon a computer readable description of hardware for implementing a deep neural network (DNN) and comprising an activation module for performing an activation function, the activation module having a programmable lookup table for storing lookup data representing the activation function, wherein in use, the activation module is configured to load first lookup data into the lookup table, the first lookup data being generated over a determined range of input values ​​to the activation module for performing the activation function, wherein the range of input values ​​to the activation module is determined by monitoring inputs to the activation module in a representation of the hardware, the activation module being arranged to operate on calibration data; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit implementing the hardware; and An integrated circuit production system is configured to manufacture the hardware according to the circuit layout description.

Images