Neural Engine

Abstract

The present invention is entitled "Neural Network Engine". The present invention discloses a neural network engine, which includes a plurality of floating-point multipliers, each having an input connected to an input mapping value and an input connected to a corresponding kernel value. The multiplier pairs provide outputs to a tree of nodes, each node of the tree being configured to provide a floating-point output corresponding to either: the larger of the inputs of the node; or the sum of the inputs, an output node of the tree providing a first input to an output module, and one of the multipliers providing an output to a second input of the output module. The engine is configured to process any convolutional layer of the neural network, i.e., an average pooling layer or a maximum pooling layer, based on the kernel value and whether the node and output module are configured to output the larger of its inputs or the sum of its inputs.

Description

Technical Field

[0001] The present invention relates to a neural network engine. Background Art

[0002] Figure 1 The processing flow of a typical convolutional neural network (CNN) is presented in . Typically, the input to the CNN is at least one 2D image / map 10 corresponding to a region of interest (ROI) from an image. The image / map may include only image intensity values, such as the Y plane from a YCC image; or the image / map may include any combination of color planes from the image; or alternatively or in addition, the image / map may contain values ​​derived from the image, such as a Histogram of Gradients (HOG) map or an integral image map as described in PCT Application No. PCT / EP2015 / 073058 (Reference: FN-398).

[0003] CNN processing consists of two stages:

[0004] ○ Feature extraction (12) - the convolution part; and

[0005] ○Feature classification (14).

[0006] CNN feature extraction 12 typically includes multiple processing layers 1...N, where:

[0007] Each layer consists of a convolution followed by optional subsampling (pooling).

[0008] Each layer produces one or (usually) multiple maps (sometimes called channels);

[0009] The size of the map after each convolutional layer is usually reduced by subsampling (examples of subsampling are average pooling or max pooling);

[0010] ● The first convolutional layer typically performs a 2D convolution of the original 2D image / map to produce its output map, while subsequent convolutional layers perform 3D convolutions using the output map produced by the previous layer as input. However, if the input includes multiple maps previously derived from an image, for example; or

[0011] If the input image / map contains multiple color planes, such as RGB or YCC; or multiple versions of an image, the first convolutional layer can operate in exactly the same way as successive layers, performing a 3D convolution on the input image / map.

[0012] Figure 2 Shown by Figure 1 An exemplary 3D convolution with a 3×3×3 kernel performed by the subsequent feature extraction convolutional layer of . 3×3×3 means that three input maps A, B, C are used, so a 3×3 pixel block from each input map is required to compute one pixel in the output map.

[0013] The convolution kernel also has 3×3×3=27 values ​​or weights that are pre-computed during the training phase of the CNN. The cube of input mapped pixel values ​​16 is combined with the convolution kernel values ​​18 using a dot product function 20. After the dot product is calculated, an activation function 22 is applied to provide an output pixel value. The activation function 22 may include a simple division (as is typically done in convolution), or a more complex function such as a sigmoid function or a rectified linear unit (ReLU) activation function of the form: j =h(x j )=max(0,x j ) (as commonly used in neural networks).

[0014] In this case, for a 2D convolution using a single input image / map, the input image / map will be scanned with a 3×3 kernel to produce the pixels of the corresponding output map.

[0015] Within a CNN engine such as that disclosed in PCT Application No. PCT / EP2016 / 081776 (reference: FN-481-PCT), a processor needs to efficiently implement the logic required to perform processing of different layers such as convolutional layers and pooling layers. Summary of the invention

[0016] According to the present invention, a neural network engine is provided.

[0017] Embodiments of the present invention incorporate a module for outputting floating point results of pooling or convolutional layer operations in a neural network processor.

[0018] By sharing the resources required for both layer types, the processing pipeline within the Neural Engine can be kept simple and the number of logic gates as low as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0020] Figure 1 A typical convolutional neural network (CNN) is shown;

[0021] Figure 2 An exemplary 3D convolution with a 3×3×3 kernel is shown;

[0022] Figure 3 A typical floating point adder is shown;

[0023] Figure 4 A tree for implementing a 3×3 convolution kernel is shown;

[0024] Figure 5(a) and Figure 5(b) show the trees for 3×3 averaging and max pooling, respectively;

[0025] Figure 6 shows processing logic configured to implement a convolutional network layer according to one embodiment of the present invention;

[0026] Figure 7 shows a configuration for implementing an average pooling layer Figure 6 The processing logic;

[0027] Figure 8 shows a configuration for implementing a maximum pooling layer Figure 6 The processing logic;

[0028] Fig. 9 The conventional ReLU activation function is shown;

[0029] Fig.10 shows a PReLU activation function implemented by one embodiment of the present invention;

[0030] Fig.11 More details are shown for implementing Fig.10 The interface of the PReLU activation function Figures 6 to 8 Output module;

[0031] Fig.12 Conventional PEAK operation is shown; and

[0032] Fig.13 The example shown is used to implement an additional PEAK operation or two parallel 2×2 max pooling operations. Figures 6 to 8 Variants of the processing logic. DETAILED DESCRIPTION

[0033] Floating point (FP) representations are commonly used for convolution kernel weights and image / map values ​​processed by a CNN engine. The typical format suitable for hardware CNN engine implementations is 16-bit, half-precision floating point (IEEE 754-2008). However, 8-bit FP representations can also be used, and PCT Application No. PCT / EP2016 / 081776 (reference: FN-481-PCT) discloses how the default FP exponent bias can be changed to a custom value to avoid the need to use a higher precision FP format.

[0034] Figure 3 An exemplary conventional 16-bit FP (IEEE 754-2008) floating point adder is shown - it is clear that the structure is similar for any precision. As is well known, such an adder must know which of the two operands, s1 / exp1 / m1 and s2 / exp2 / m2 respectively, is greater in order to align the mantissas of the operands. Therefore, the adder shown includes a comparator 40 that produces a binary output (O / P) indicating which of the floating point operands is greater.

[0035] The sign bits s1, s2 of these operands are then used to select between the following two versions: a version of m1 shifted according to the difference between exp2 and exp1 subtracted from m2; or a shifted version of m1 plus m2; and to select between the following two versions: a version of m2 shifted according to the difference between exp2 and exp1 subtracted from m1; or a shifted version of m2 plus m1 by multiplexer 44. Multiplexers 46, 48 and 50 are now used according to the output O / P of comparator 40, and the output of either multiplexer 42 or 44 is selected to generate the mantissa of the output of the adder, the value of the mantissa is also used to adjust whichever of exp1 or exp2 is selected by multiplexer 48 to generate the exponent of the output of the adder, and the sign of the output of the adder is selected to be s1 or s2.

[0036] It should be noted that this adder is shown for exemplary purposes only and that the present invention is equally applicable to any equivalent floating point adder.

[0037] See now Figure 4 , we can see that the convolutional layer needs to: implement the logic of the floating-point multiplier (30) and implement Figure 2 The adder logic (32) for the dot product function (20) is shown, as well as the logic for the activation function (22). It should be noted that any 3D convolutional layer can be decomposed into a longer chain of multipliers, and then its products are added in a deeper tree than shown. Therefore, Figure 2 The 3×3×3 convolution shown may require 27 multipliers. Figure 4 The logic shown can therefore be extended to handle the largest form of convolution that would be performed by a given CNN engine, and where a smaller or simpler 2D convolution would be performed, and then, for example, selected invalid weights within the kernel could be zeroed.

[0038] Separately, Figure 5(a) shows that the average pooling layer requires: an adder (34) and logic for implementing the multiplication by 1 / kernel size (36); while Figure 5(b) shows that the max pooling layer requires a comparator (38).

[0039] observe Figure 4 5, it can be seen that the adder (32, 34) and comparator (38) trees have a similar structure, where each node in the tree produces the sum of its inputs or the maximum of its inputs, respectively.

[0040] It can also be seen that any floating-point adder (such as Figure 3Adders of this type already include comparator circuitry 40 for determining which of its operands is greater. Thus, with little additional configuration circuitry, any such adder can be converted to select, based on a single control input, whether the sum of its input operands (as normal) or the greater of the operands is fed to the comparator. The logic paths to produce these results are separate, so this functionality can be implemented without affecting the timing of the adder.

[0041] See now Figure 6 , shows a circuit 70 according to one embodiment of the present invention, which is used to enable a CNN engine to process convolution, maximum pooling or average pooling layers of a neural network. The circuit 70 includes a conventional floating point multiplier 30 (such as Figure 4 Currently, no such Figure 4 Rather than the tree of floating point adder nodes 32 shown, a tree 32' of configurable nodes is provided which can be set to produce the sum of their inputs or the maximum of their inputs depending on the value of the Op_sel control input.

[0042] In this embodiment, the output of the final node in the tree 32' and the output of one of the multipliers 32 are fed to the output module 60 as In0 and In1, respectively.

[0043] Circuit 70 is configured to operate in a number of different modes:

[0044] exist Figure 6 , the tree 32' is configured to perform a 3x3 2D convolution by causing the nodes to perform an addition, i.e., Op_sel is set to indicate addition and fed to each node. The now trained kernel weight values ​​18 are combined with the input image / map values ​​16 (typically around a given pixel corresponding to in11 in the multiplier 30) as required before being fed through the tree 32' to the output module 60. The activation function enables the signal to be fed to the output module 60, and this will cause the output module 60 to apply the required activation function to the result of the operation (in this case, addition) on In0 and In1 determined by the value of Op_sel before providing the convolution output for the given pixel of the input image / map, as will be explained in more detail below.

[0045] Figure 7 The same tree 32' is shown to be configured for average pooling by setting the weight value 18 in the kernel to 1 / 9, where 9 is the number of (valid) units in the kernel, and having nodes in the tree 32' configured to perform addition. In this case, the output module 60 only needs to add In0 and In1 according to the value of Op_sel to provide its output. Therefore, it is only necessary to apply the inactive function (or simple identity function) to the results of the operations of In0, In1.

[0046] at last, Figure 8 It is shown that the same tree 32' can be configured for maximum pooling in the following manner: the (effective) weight value 18 in the kernel is set to 1, and the tree 32' has nodes configured only as comparators to produce the maximum value of its operands. In this case, the output module 60 only needs to select the larger of In0 and In1 as its output according to the value of Op_sel. Similarly, it is only necessary to apply the inactive function (or simple identity function) to the operation results of In0 and In1.

[0047] Again, the tree presented here is just one example for a 3×3 kernel. Other kernel sizes can be easily implemented.

[0048] As indicated, convolution requires an activation function 22 to be applied to the dot product 20 of the input image / map values ​​16 and the weight kernel values ​​18. Currently, the most commonly used activation functions are the identity function (i.e., no activation function is applied), ReLU, and PReLU.

[0049] like Fig. 9 As shown, ReLU (rectified linear unit, which is a very simple function) is defined as follows:

[0050] ReLU(x) = x, where x>0

[0051] ReLU(x)=0, where x<=0

[0052] PReLU (parameterized ReLU, which is a more general form of the ReLU function) is defined as follows:

[0053] pReLU(x) = x, where x>0

[0054] pReLU(x) = x / slope_coef, where x <= 0

[0055] Rather than using the exact slope, some embodiments of the invention use only slopes that are powers of 2. While such slopes can be used as approximations to the specified slope, this approach works best if the network is trained using the approximated PReLU function, rather than approximating the PReLU function after training.

[0056] In any case, this approximation requires two inputs:

[0057] ●Coef - logarithm of the slope coefficient

[0058] = PSlope - Flag indicating whether the slope is negative or positive.

[0059] Therefore, see Fig.10 , the PReLU function becomes:

[0060] pReLU(x) = x, where x>0

[0061] pReLU(x)=x / (2^Coef), where x<=0 and PSlope=0

[0062] pReLU(x) = -x / (2^Coef), where x <= 0 and PSlope = 1

[0063] Looking at the approximated PReLU activation function, it can be seen that it only needs to modify the exponent of the input.

[0064] See now Fig.11 And based on the observations and approximations discussed above, the output module 60 can implement the addition, maximum, ReLU and PReLU functions as needed. The Op_sel input is used to select between addition or maximum of the two operands In0 and In1.

[0065] The activation input may have two bits and may select between three activation types as shown in Table 1. It will therefore be appreciated that one or more activation functions may be added, in particular by increasing the number of control bits.

[0066]

[0067] Table 1

[0068] For PReLU, these coefficients can be encoded using 4 bits, representing the logarithm of the coefficient:

[0069] 4-bit PReLU Coef PReLU slope coefficient 0000 1 0001 2 0010 4 … … 1111 2^15

[0070] Output module 60 may utilize a floating point pipeline to perform the division by a power of two coefficient by subtracting the 4-bit PReLU coefficient instruction field value from the FP16 exponent value of the sum of In0 and In1 (or possibly the larger of these two values, as explained below).

[0071] Using this scheme, the identity activation function can be implemented as a PReLU form with PSlope=0 and Coef=0000, allowing the activation input to be defined with a single binary activation input with settings ReLU=0 or PReLU=1, so Fig.10 The PReLU function shown can be modified as follows:

[0072] pReLU(x) = x, where x>0

[0073] pReLU(x) = (x / (2^Coef), where x <= 0 and PSlope = 0) * activation input

[0074] pReLU(x) = (-x / (2^Coef), where x <= 0 and PSlope = 1) * activation input,

[0075] Thus enabling the function to be implemented in a single pipeline.

[0076] The subnormal / infinity special case can be handled by implementing the logic of the PReLU activation function as follows:

[0077] ● If the input is negative, NaN, or infinite, the input value is passed to the output unchanged;

[0078] ●If the result is below normal, suppress the result to 0.

[0079] It should be understood that, typically, only activation functions other than the identity function will be used when processing convolutional layers. However, using activation functions such as Fig.11 When the output module 60 with configuration control is shown, a non-linear activation function can also be applied to the pooling result. Although uncommon, this option can be useful in some cases.

[0080] See now Fig.12 Another type of layer that can be incorporated into the network may involve detecting peaks within the image / map. The PEAK operation uses an M×N window ( Fig.12 The source image / map is scanned in raster order with a given step size (e.g. 1) and produces a map containing a list of found peaks - a peak is an image / map element whose value is strictly greater than any surrounding elements in the window.

[0081] See also Fig.13 , implementing the PEAK operation also requires that the nodes of the tree 32' be configured as comparators, which is the same as the case of maximum pooling. The only condition is that the middle element of the M×N window (corresponding to the pixel position of the image / map to be processed) is connected to a multiplier, the output of which is directly fed to In1 of the output module 60', i.e. Fig.13 In the example shown, in11 and in22 are swapped. The output module 60' now produces two outputs, one output indicating a peak value (in terms of max pooling) and a second output indicating whether In1 is greater than In0, ie whether In1 is a valid peak.

[0082] One way to provide this second output is for the second output to include a binary value indicating the maximum operand, such as In0 being 0 or In0 being 1. Such an output may also apply to functions other than PEAK.

[0083] In a further variant, it is also possible to have the tree 32' provide the results of two pooling operations simultaneously. Fig.13In the example of , the tree 32' generates two 2x2 max pooling operations in parallel. In this case, the outputs are taken from two adders 32(1), 32(2) in the penultimate level of the tree. If desired, these outputs may also be fed through an activation function, but in any case this allows the step size of the pooling operation to be increased, thereby reducing the processing time of the image / map. Again, this may require a reorganization of the input image / map values ​​so that the required 2x2 sub-window from the image value 16 is fed to the appropriate node of the tree 32' via the multiplier 30.

[0084] It can also be seen that Fig.13 The illustrated 2×2 pooling method may be adapted to provide average pooling by setting the weight values ​​18 to 1 / 4 and configuring the nodes of the tree 32 ′ for summation.

[0085] It will be appreciated that where processing is only being used for windows of a particular size for 2D or 3D convolution or pooling, the connections from the input image / map and kernel values ​​16, 18 to the multiplier 30 may be hard-wired. However, where the window size may vary, then, particularly for peak layer processing and sub-window pooling, a multiplexer circuit (not shown) will need to be inserted between the engine's inputs 16, 18 and the multiplier 30.

[0086] The above implementation can be implemented using several variations of the basic floating point adder:

[0087] ● Normal FP adders used in output modules 60, 60';

[0088] ● FP adder with activation function used in output modules 60, 60';

[0089] - combined adders and maximums used within tree 32'; and

[0090] • Combined adder and maximum with activation function used within output modules 60, 60'.

[0091] Using these variants of adders, commonly used engines (such as the CNN engine disclosed in PCT Application No. PCT / EP2016 / 081776 (Reference: FN-481-PCT)) can be reconfigured for convolution, pooling, or peak operations with minimal additional logic compared to implementing any given layer independently.

Claims

1. A hardware neural network engine, the neural network engine configured to receive at least one M×N window of floating point values ​​corresponding to pixels of an input map and a corresponding at least one set of M×N floating point kernel values ​​of a neural network layer of a neural network, wherein each of M and N is an integer value, the neural network engine comprising: a plurality of at least M×N floating point multipliers (30), each floating point multiplier having a first operand input configured to be connected to an input mapping value (in00, ..., in 22), and a second operand input configured to be connected to a corresponding kernel value (w00, ..., w22); a pair of multipliers within the M×N floating point multipliers (30), the pair of multipliers providing respective floating point outputs to respective input nodes of a tree (32') of nodes, one output node of the tree providing a first input (In0) of an output module (60), and an output of one of the M×N floating point multipliers providing a second input (In1) of the output module (60); wherein when the engine is configured to process a convolutional layer of the neural network, each of the kernel values ​​comprises a training value for the layer, the nodes of the tree are configured to sum their inputs, and the output module (60) is configured to sum its first input and second input, apply an activation function to the sum of the first input and the second input and provide an output of the activation function (22) as an output of the output module; Features: Each node of the tree is configured to provide a floating point number output corresponding to either: the greater of the inputs to the node; or the sum of the inputs; and wherein when the engine is configured to process an average pooling layer of the neural network, each of the kernel values ​​comprises a value corresponding to 1 / (M×N), the nodes of the tree (32′) are configured to sum their inputs, and the output module is configured to sum its first input and second input and provide the sum as the output of the output module; and Wherein when the engine is configured to process the maximum pooling layer of the neural network, each of the kernel values ​​comprises a value equal to 1, the nodes of the tree (32') are configured to output the larger of their inputs, and the output module is configured to output the larger of its first input and second input as the output of the output module.

2. A neural network engine according to claim 1, wherein the activation function is one of the following: an identity function, a ReLU function or a PReLU function.

3. A neural network engine according to claim 1, wherein the activation function is defined by a binary slope parameter and a slope coefficient.

4. The neural network engine according to claim 3, wherein the activation function PReLU(x) is defined as follows: PReLU(x) = x, where x>0 PReLU(x)=x / (2^Coef), where x<=0 and PSlope=0 PReLU(x)=-x / (2^Coef), where x<=0 and PSlope=1, Wherein Coef is the slope coefficient and PSlope is the binary slope parameter; wherein x is the sum of the first input and the second input.

5. A neural network engine according to claim 4, wherein the engine is configured to implement the activation function by modifying only the exponent of the input x.

6. A neural network engine according to claim 1, wherein when the engine is configured to process a peak-finding layer of the neural network, each of the kernel values ​​comprises a value equal to 1, the nodes of the tree are configured to output the larger of their inputs, and the output module is configured to output the larger of its first input and second input as the output of the output module, and wherein the engine is configured to provide an image value from within the M×N window corresponding to the pixel as an input to the one of the M×N floating-point multipliers, and provide the output of the one of the M×N floating-point multipliers to the second input of the output module.

7. A neural network engine according to claim 6, wherein the output module is configured to provide a second output, and the second output indicates whether the second input of the output module is greater than the first input of the output module.

8. A neural network engine according to claim 1, wherein the engine is further configured to provide multiple outputs from the penultimate node (32(1), 32(2)) of the tree so that when the engine is configured to process a pooling layer, the engine simultaneously provides multiple pooling results for corresponding sub-windows of the M×N window.

Images