Configurable non-linear activation function circuit

Abstract

Certain aspects of the present disclosure provide a method for processing input data by a set of configurable non-linear activation function circuits, including generating an exponential output by processing input data using one or more first configurable non-linear activation function circuits configured to execute an exponential function; summing exponential outputs of the one or more first configurable non-linear activation function circuits; and generating an approximate logarithmic softmax (soft maximum) output by processing the summed exponential output using a second configurable non-linear activation function circuit configured to execute a natural logarithmic function.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 18 / 165,802, filed on February 7, 2023, which is a continuation-in-part of U.S. patent application No. 17 / 467,079, filed on September 3, 2021, the entire contents of each of which are incorporated herein by reference.

[0003] introduction

[0004] Aspects of the present disclosure relate to processing nonlinear activation functions for machine learning models, and more particularly to configurable nonlinear activation function circuits.

[0005] Machine learning is generally the process of producing a trained model (e.g., an artificial neural network) that represents a generalized fit to a training data set. Applying the trained model to new data enables the production of inferences, which can be used to gain insights about the new data.

[0006] As the use of machine learning has surged to implement various machine learning (or artificial intelligence) tasks, there has been a demand for more efficient processing of machine learning model data. In some cases, dedicated hardware (such as a machine learning (or artificial intelligence) accelerator or processor or similar circuitry) can be used to enhance the ability of a processing system to process machine learning model data. For example, processing data with nonlinear activation functions can be distributed to processors other than the main matrix multiplication processor. However, distributing various aspects of processing a machine learning model across different processing devices can result in latency, memory usage, power consumption, and other processing losses.

[0007] Therefore, improved techniques are needed to handle data for machine learning models with nonlinear activation functions.

[0008] Brief Overview

[0009] Certain aspects provide a processor comprising: one or more first configurable nonlinear activation function circuits configured to perform an exponential function on input data; a summation circuit configured to receive output data of the one or more first configurable nonlinear activation function circuits; and a second configurable nonlinear activation function circuit configured to receive the output data of the summation circuit, perform a natural logarithm function, and output an approximate logarithm softmax of the input data.

[0010] A further aspect provides a method for processing input data through a set of configurable nonlinear activation function circuits, comprising: generating an exponential output by processing the input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function; summing the exponential outputs of the one or more first configurable nonlinear activation function circuits; and generating an approximate logarithmic softmax output by processing the summed exponential output using a second configurable nonlinear activation function circuit configured to perform a natural logarithm function.

[0011] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising means for performing the aforementioned methods and those further described herein.

[0012] The following description and the associated annexed drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings depict certain of the one or more aspects and, therefore, are not to be considered limiting of the scope of the disclosure.

[0015] Figure 1 An example configurable nonlinear activation (CNLA) function circuit is depicted.

[0016] Figure 2 Depicts example circuit blocks for implementing a bypassable approximator block, such as with respect to Figure 1 described.

[0017] Figure 3 An example approximator is depicted.

[0018] Figure 4 Depicts an example machine learning model process flow.

[0019] Figure 5 Depicted are example methods for performing processing using configurable non-linear activation function circuits.

[0020] Figure 6 Depicts an example architecture of a CNLA function circuit for performing a softmax operation using parallel input data.

[0021] Figure 7Depicts an example architecture of a CNLA function circuit for performing a softmax operation using sequential input data.

[0022] Figure 8 Depicted is an example method for performing an approximate softmax operation using a CNLA function circuit.

[0023] Figure 9 An example processing system is depicted that may be configured to perform the methods described herein.

[0024] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation.

[0025] Detailed description

[0026] Aspects of the present disclosure provide improved techniques for handling non-linear activation functions associated with machine learning models.

[0027] Nonlinear activation is a key component of various types of machine learning models, including neural network models. While some nonlinear activation functions are implemented as piecewise linear functions (e.g., rectified linear units (ReLU), leaky ReLU, etc.), other nonlinear activation functions require complex mathematical functions (e.g., sigmoid, hyperbolic tangent (tanh), etc.). In some cases, the complex mathematical functions can be implemented using interpolation (such as cubic spline interpolation). For example, the interpolated output value can be determined in some aspects using a lookup table (LUT) to match the output value to the input value. When the target input value is not mapped in the LUT, the LUT value associated with the input value adjacent to the target input value can be used to interpolate the output value for the target input value.

[0028] Conventionally, due to the wide range of possible activation functions available in machine learning models, nonlinear activation functions are implemented in software rather than hardware. However, such implementations typically require moving model data between processing devices (e.g., between a neural processing unit (NPU) that performs matrix multiplication and accumulation and a digital signal processor (DSP) that handles the nonlinear activation functions), thereby incurring power and latency penalties. In cases where nonlinear activation functions have been implemented in hardware, they are typically limited to supporting only a small number of nonlinear activation functions and therefore cannot be configured to support evolving machine learning model architectures without falling back to outsourcing the nonlinear activation function processing to other processing units.

[0029] For example, the rectified linear unit (ReLU) is a commonly used activation function in deep learning models. If a negative input is received, the function returns 0, otherwise it returns the input x. Therefore, it can be written as f(x) = max(0,x). The ReLU function is not typically implemented by a main matrix multiplication and accumulation processing unit, such as a compute-in-memory (CIM) array in some examples. Therefore, from a processing perspective, the need to distribute the ReLU function or another nonlinear activation function can be expensive. In addition, as the activation function becomes more complex, the processing cost also becomes more significant (for example, for performing relatively high exponential and division operations as part of certain nonlinear activation functions, as described below).

[0030] To overcome the shortcomings of conventional solutions, various aspects described herein relate to a configurable nonlinear activation (CNLA) function circuit that can be implemented in hardware for efficient processing. In particular, because it can be implemented in hardware, the CNLA function can be co-located with other processing circuits optimized for other machine learning model processing tasks, such as CIM arrays and digital multiply and accumulate (DMAC) circuits optimized for performing vector and matrix multiplication and accumulation functions.

[0031] In order to improve processing efficiency, various aspects described herein can use polynomial approximation to approximate complex functions, such as can be used in nonlinear activation functions. In some cases, various aspects described herein can use series expansion, such as Taylor series. Usually, the Taylor series of a function (e.g., f(x)) is an infinite sum of terms represented by the derivative of the function at a single point. For many functions, the sum of the function and its Taylor series is equal near this point. The partial sum formed by the first n+1 terms of the Taylor series is an n-order polynomial, which is called the n-th Taylor polynomial of the function. Therefore, Taylor polynomials allow efficient approximation of processing functions, which usually becomes better as n increases.

[0032] The CNLA function circuit described herein can implement one or more polynomial approximation blocks, such as a cubic approximation block, which typically enhances cubic spline interpolation to make it more efficient and more general, thereby covering a wider range of nonlinear activation functions. In addition, the CNLA function circuit can be implemented as a pipelined digital block that can use nonlinear piecewise lookup tables (LUTs) and mixed-order approximations (e.g., pipelined linear, quadratic, and cubic approximations). Therefore, unlike conventional nonlinear activation function circuits, the CNLA function circuit described herein can be configured to meet many different performance targets.

[0033] Accordingly, the CNLA function circuits described herein provide a technical solution to the technical problem of implementing a wide range of nonlinear activation functions in machine learning model processing systems. Furthermore, compared to existing solutions, the CNLA function circuits described herein provide technical improvements through increased model processing performance, including lower latency, lower power usage, improved memory efficiency, and other benefits described herein.

[0034] Example configurable nonlinear activation function circuit

[0035] Figure 1 An example configurable non-linear activation (CNLA) function circuit 100 is depicted.

[0036] In general, the CNLA function circuit 100 can be configured to receive input data 101 (e.g., output values ​​from a layer of a machine learning model) and perform various non-linear activation functions to generate output data 114 (e.g., "activations"). The CNLA function circuit 100 can be co-located and pipelined with other machine learning model processing circuits (such as a CIM array, a DMAC, etc.) and can be configured to perform activation functions based on the outputs of the other machine learning model processing circuits.

[0037] In some examples, the input data 101 may be received from a buffer or other memory. In other examples, the input data 101 may be received directly from the output of another processing block, such as a CIM array or the output of another vector and matrix multiply and accumulate block.

[0038] The CNLA function circuit 100 includes a first approximator block 102, which may generally be configured to perform a hardware-based mathematical function, such as a first approximator block 102, on input data 101. Figure 3 An example approximator is described in detail.

[0039] In some cases, the first approximator is a linear approximator (for example, configured to perform a linear function such as ax+b), a quadratic approximator (for example, configured to perform a quadratic function such as ax+b), or a quadratic approximator (for example, configured to perform a quadratic function such as ax+b). 2 +bx+c) or a cubic approximator (e.g., configured to perform a cubic function such as ax 3 +bx 2+cx+d), where x is the input data and a, b, c, and d are configurable parameters. In general, a linear, quadratic, or cubic approximator can be used to approximate a given function, which may or may not be a polynomial function. The first approximator 102 can be configured with parameters retrieved from, for example, a memory, a register, a lookup table, etc. As described in further detail below with respect to Table 2 below, these different forms of approximation and the associated configurable parameters can be used to approximate many types of nonlinear activation functions.

[0040] CNLA function circuit 100 further includes a second approximator block 104, which, similar to first approximator block 102, can generally be configured to perform hardware-based mathematical functions, such as linear, quadratic, or cubic functions. As described in more detail below, CNLA function circuit 100 can be configured to connect first approximator block 102 and second approximator block 104 in series for more complex functions, such that the output of first approximator block 102 becomes the input of second approximator block 104. CNLA function circuit 100 can also be configured to use only one of first approximator block 102 or second approximator block 104 when processing simpler nonlinear functions, thereby saving power.

[0041] In some implementations, first approximator 102 and second approximator 104 may include the same circuit block (e.g., two instances of the same circuit element within circuit 100). For example, in some aspects, first approximator 102 and second approximator 104 may include cubic approximators. In other implementations, first approximator 102 and second approximator 104 may include different circuit elements, and in such cases, typically second approximator 104 will include a cubic approximator and first approximator 102 will include a lower-order approximator, such as a quadratic or linear approximator. However, in other aspects, the orders of the high-order and low-order approximators may be reversed.

[0042] CNLA function circuit 100 includes a configurable bypass 105 that allows first approximator 102 to be bypassed in various scenarios, such as when the function only requires an approximator of a lower order than first approximator 102, and second approximator 104 is such a lower order approximator. For example, when first approximator 102 is bypassed via configurable bypass 105, input data 101 is instead provided directly to second approximator 104 without being processed by first approximator 102. In various aspects, first approximator 102 can be a higher order approximator than second approximator 104, or vice versa, or they can be of the same order (e.g., both are linear, quadratic, or cubic). Configurable bypass 105 allows for saving processing time and energy when only one approximator is required.

[0043] CNLA function circuit 100 further includes another configurable bypass 107 that allows second approximator 104 to be bypassed in various scenarios, such as where the function only requires a first approximation and first approximator 102 is capable of performing the first approximation in the absence of second approximator 104. For example, when second approximator 104 is bypassed via configurable bypass 107, the output of first approximator 102 is provided directly to multiplier 108.

[0044] In general, configurable bypasses 105 and 107 allow the CNLA function circuit 100 to be configured for maximum versatility while saving power and avoiding unnecessary circuit block processing in a variety of scenarios. In addition, the configurable bypasses allow asymmetric and antisymmetric nonlinear activation functions to be configured to be processed by the CNLA function circuit 100. Figure 2 Various example circuit aspects for implementing configurable bypasses 105 and 107 (eg, bypasses 205A and 205B) are depicted.

[0045] The CNLA function circuit 100 further includes a gain block 106 configured to provide a gain value to the multiplier 108. In some aspects, the gain block 106 is configured to generate a gain value 109 based on a gain function implemented by the gain block 106. In one example, the gain function can be of the form g=ax+b, where g is the gain value, x is the input data 101 value, and a and b are configurable parameters. More generally, the gain block 106 can multiplicatively (a) and / or additively (b) modify the input data to generate a gain value.

[0046] The gain value 109 generated by the gain block 106 is multiplied by the output of the first approximator 102 and / or the second approximator 104 via the multiplier 108. In other aspects, the gain block 106 can be configured to have a gain value that is not based on a function of the input data 101 (e.g., by setting a to zero in the expression for g above). Typically, the parameters (e.g., a and b in the above example) or values ​​of the gain block 106 can be retrieved from, for example, a memory, a register, a lookup table, etc.

[0047] CNLA function circuit 100 further includes a constant block 110 configured to store a configurable (e.g., programmable) constant value 113 and an adder 112 configured to add the constant value 113 to the output of multiplier 108 (e.g., a gain multiplier). The constant value 113 stored in constant block 110 can be retrieved from, for example, a memory, a register, a lookup table, etc.

[0048] The inclusion and arrangement of the first approximator block 102, the second approximator block 104, configurable bypasses 105 and 107, the gain block 106, the multiplier 108, the constant block 110, and the adder 112 allow the CNLA function circuit 100 to be configured to perform a variety of known and later developed nonlinear activation functions. Furthermore, the CNLA function circuit 100 can be efficiently configured to handle a wide variety of nonlinear activation functions by simply updating the parameters of the first approximator 102, the second approximator 104, the gain block 106, and the constant block 110. When two approximator blocks 102 and 104 are used together to simulate a nonlinear function, each approximator block 102 and 104 can be said to perform a corresponding individual function (e.g., a first function performed by the first approximator block 102 and a second function performed by the second approximator 104). This design advantageously supports arbitrary asymmetric nonlinear curves for complex functions.

[0049] Table 1 below provides the Figure 1 The CNLA function circuit 100 can be configured to perform various nonlinear activation functions, including example parameters for Figure 2 In Table 1, the gain is considered to have the form ax + b, as in Figure 1 , but note that in other aspects, the gain can be a scalar value, or a different functional form. Similarly, the quadratic approximator is considered to have ax 2 +bx+c, the cubic approximator is considered to have ax 3 +bx 2 In the following table, subscripts are used to indicate parameter assignments, for example, G for the gain parameter, 1 for the first approximator, and 2 for the second approximator parameter.

[0050] Table 1

[0051]

[0052]

[0053]

[0054]

[0055] Note that in the ELU function above, the α parameter is configurable as a hyperparameter by the model designer.

[0056] It is worth noting that in some implementations, the parameters of an approximator can be given in a certain form (e.g., cubic with a, b, c, and d parameters or quadratic with a, b, and c parameters) even if the approximator is performing a lower-order function (e.g., linear). This is because, for example, setting the cubic parameter a to zero effectively collapses the approximation equation into a lower-order quadratic function, and similarly, setting the quadratic parameter a to zero effectively collapses the approximation equation into a linear equation. Thus, when an approximator is configured with quadratic parameters, the approximator can be configured for a "quadratic function," for example, but the result of these parameters can reduce the function to a linear function, as in the example of ReLU in Table 2 above. This can allow for standardization of parameter sets regardless of the order of the underlying functions to be configured by the parameters, thereby simplifying implementation.

[0057] Figure 2 Depicted are example circuit blocks 202 and 204 for implementing bypassable approximator blocks 206A and 206B. In one example, bypassable approximator blocks 206A and 206B may correspond to Figure 1 The first approximator block 102 and the second approximator block 104 are described in detail.

[0058] exist Figure 2 , circuit block 202 is configured to control the use of function block 214A, which in this example includes a first approximator 206A and a minimum and maximum function block 208A. Similarly, circuit block 204 controls the use of function block 214B, which in this example includes a minimum and maximum function block 208B and a second approximator 206B. The first and second approximator blocks 206A and 206B can be configured to implement nonlinear activation functions, such as those described above with respect to Table 1.

[0059] Notice, Figure 1 The first approximator 102 in FIG. 1 requires only one input, but the circuit block 202 includes two input ports 201A and 201B that allow for multiple inputs. The depicted configuration of the circuit block 202 can be employed so as to present the same external interface to both circuit blocks 202 and 204, which can simplify configuration and integration. In some aspects, the circuit block 202 receives a single input (such as Figure 1 In an implementation where the input data 101 is received from the circuit block 202, the two input ports 201A and 201B of the circuit block 202 can be tied together. In an alternative implementation, the circuit block 202 can be simplified by removing the input port 201B and removing the input multiplexer (mux) 203A so that 201A will be directly provided to 214A and 207A.

[0060] Generally, input ports 201A and 201B can receive various types of input data for processing, including signed multi-bit integer data. In one example, the input data is 8-bit 2's complement input data.

[0061] Input selector multiplexers 203A and 203B are configured to control which input data port is used for circuit blocks 202 and 204, respectively. For example, input selector multiplexer 203B may select between input data port 201A (e.g., when circuit block 202 is bypassed) or 212B (e.g., when circuit blocks 202 and 204 are processed in series).

[0062] Bypass selector multiplexers 211A and 211B are configured to control bypassing of function blocks 214A and 214B of circuit blocks 202 and 204, respectively. For example, when circuit block 202 is to be bypassed, bypass selector multiplexer 211A selects bypass 205A to provide an output to output port 212A. Similarly, when circuit block 204 is to be bypassed, bypass selector multiplexer 211B selects bypass 205B to provide an output to output port 216. Thus, processing by circuit blocks 202 and / or 204, controlled by configurable bypasses 205A and 205B, results in an output at output port 216.

[0063] As about Figure 3 As discussed in more detail, approximator blocks 206A and 206B may be configured with configuration parameters (e.g., function-specific coefficients as in Table 1 above) stored in registers 219A and 219B, respectively. Similarly, as in Table 1 above, where approximator block 206A or 206B is configured to perform a lookup table-based function, the table values ​​may be stored in registers 219A and 219B, respectively.

[0064] Each circuit block (202 and 204) further includes a minimum and maximum function block (208A for circuit block 202 and 208B for circuit block 204) for providing minimum and maximum functions. Generally, a minimum (or "min") function will return the minimum value of the input provided. Similarly, a maximum (or "max") function will return the maximum value of the input provided. In one example, the minimum and maximum function blocks 208A and 208B may include multi-bit digital comparators operating in single-cycle or multi-cycle mode.

[0065] The configuration of function blocks 214A and 214B may include settings for function selector multiplexers 209A and 209B, respectively. In other words, whether function blocks 214A and 214B output the minimum / maximum output from min / max blocks 208A and 208B or the value from approximators 206A and 206B is based on the configuration of function selector multiplexers 209A and 209B. Note that in other examples, function blocks 214A and 214B may include additional function blocks that can be selected by the multiplexers.

[0066] like Figure 1 As depicted in , where the approximator blocks can be processed in series, Figure 2 , the output 212A of the circuit block 202A including the first approximator block 206A is provided as an input 212B to the circuit block 204 including the second approximator block 206B. Figure 1 Whereas the bypasses 105 and 107 control the use of the first and second approximator blocks 102 and 104, the optional bypasses 205A and 205B control the use of the approximator blocks 206A and 206B.

[0067] Asymmetric signal line 210 controls the configuration of circuit blocks 202 and 204. In one example, circuit blocks 202 and 204 are configured based on the value on asymmetric signal line 210 and the output values ​​from sign blocks 207A and 207B based on input data received via input data port 201A. For example, the binary value received via asymmetric signal line 210 and the binary value output from sign block 207A interact at AND gate 213 to control the selection of the output by multiplexer 211A. As another example, the binary value received via asymmetric signal line 210 and the binary value output from sign block 207B interact at AND gate 217 to control the selection of the input data port (e.g., between 201A and 212B) via multiplexer 203B. As a further example, the binary value received via asymmetric signal line 210 and the inverse binary value output from sign block 207B interact at AND gate 215 to control the selection of the output by multiplexer 211B.

[0068] Table 2 below provides a summary of the configurations of circuit blocks 202 and 204:

[0069] Table 2

[0070]

[0071] Example approximator for configurable nonlinear activation function circuits

[0072] Figure 3 An example approximator 300 is depicted, which may be Figure 1 The first approximator 102 and the second approximator 104 and / or Figure 2An example of one or both of approximators 206A and 206B.

[0073] Approximator 300 receives input data 302 (e.g., pre-activation data) for processing. In some examples, input data 302 may be received from a buffer or other memory. In other examples, input data may be received directly from the output of another processing block, such as a CIM array or the output of another vector and matrix multiplication and accumulation block. Additionally, input data may be received from another approximator, such as when approximator 300 is Figure 1 The second approximator 104 and / or Figure 2 In the case of the second approximator 206B in .

[0074] In some implementations, an approximator such as 300 may include alternative processing paths. In such cases, path logic 304 may be configured to route input data 302 to the appropriate processing path based on, for example, configuration parameters of approximator 300.

[0075] In this example, processing path 306A provides a cubic approximation path for input data 302 .

[0076] In processing path 306A, input data 302 is provided to a cubic calculator 308, which performs a cubic operation (e.g., x 3 , where x is the input data), and then the output is multiplied by a cubic parameter 312 at the multiplier 310. The output of the multiplier 310 is then provided to the accumulator 324.

[0077] Input data 302 is also provided to a quadratic calculator 314, which performs a quadratic operation (e.g., x 2 , where x is the input data), and then the output is multiplied by the quadratic parameter 318 at the multiplier 316. The output of the multiplier 316 is then provided to the accumulator 324.

[0078] The input data 302 is also provided to a multiplier 320 where it is multiplied by a linear parameter 322. The output of the multiplier 320 is then provided to an accumulator 324.

[0079] An accumulator (adder) 324 accumulates the outputs of the multipliers 310 , 316 , and 320 and the intercept parameter 326 to generate output data 332 .

[0080] Cubic parameters 312, quadratic parameters 318, linear parameters 322, and intercept parameters 326 can all be stored in a memory or the like (e.g., in registers) accessible to approximator 300. In some cases, a control unit (such as a memory control unit or a finite state machine) can configure approximator 300 with the parameters stored in the memory. In various examples, cubic parameters 312, quadratic parameters 318, linear parameters 322, and intercept parameters 326 can be set according to the values ​​described above with respect to Table 2.

[0081] As described above, the order of approximation can be configured by configuring the aforementioned parameter values. For example, for an approximator 300 to perform a quadratic approximation, the cubic parameter 312 can be set to zero. Similarly, for an approximator 300 to perform a linear approximation, the cubic parameter 312 and the quadratic parameter 318 can be set to zero.

[0082] Certain nonlinear activation functions require alternative functions, such as minimum and maximum functions. Accordingly, processing path 306B provides a minimum and / or maximum calculator that can be used, for example, with the ReLU and ReLU6 functions described in Table 2 above. Processing path 306B can be selected by path logic 304 based on the configuration data of approximator 300.

[0083] In addition, certain nonlinear activation functions can be implemented using lookup tables, which provides a more efficient mechanism in terms of power and time for generating values ​​for certain nonlinear activation functions. Accordingly, processing path 306C provides a lookup table-based processing path that can be used, for example, in any case where the nonlinear activation function uses a sigmoid, tanh, or similar function. Note that sigmoid and tanh can be calculated based on each other, so in some cases, only a single lookup table (e.g., for either sigmoid or tanh, but not both) is stored and used to implement both functions. One or more lookup tables can be stored in a memory and accessible to approximator 300, including a memory tightly coupled to approximator 300.

[0084] Example machine learning model process flow using a configurable nonlinear activation function circuit

[0085] Figure 4 Describes a circuit that implements a configurable nonlinear activation function (such as the one described above for Figure 1-3 An example machine learning model data flow 400 described).

[0086] In process 400, input data is stored in input data buffer 401 (e.g., machine learning model layer input data) and then provided to multiply and accumulate (MAC) circuitry 402. MAC circuitry 402 may generally be configured to perform vector, array, and matrix multiplication and accumulation operations, such as those often used in convolutional neural networks. In some examples, MAC circuitry 402 may include one or more compute-in-memory (CIM) arrays. Alternatively or additionally, MAC circuitry 402 may include a digital multiply and accumulate (DMAC). In yet further examples, multiply and accumulate circuitry 402 may be part of a machine learning accelerator, such as a neural processing unit (NPU), or another type of processing unit optimized for performing machine learning processing. In another implementation, MAC circuitry 402 may be replaced by a vector / matrix or matrix / matrix processing engine.

[0087] The MAC circuit 402 processes the input data together with the weight data (e.g., neural network weight data) to generate pre-activation data. For example, the MAC circuit 402 may process input data to a layer of a neural network model and generate pre-activation data as output.

[0088] The pre-activation data is provided to a configurable nonlinear activation (CNLA) function circuit 404, which is configured to generate output data (e.g., activation) based on the configured nonlinear activation function. The output data can then be stored in an output data buffer 405 for subsequent use, such as for processing another layer in a machine learning model, or as output from the machine learning model.

[0089] The CNLA function circuit 404 can be configured with configuration parameters such as Figure 1 CNLA function circuit 100 and / or Figure 3 and those configuration parameters described in Tables 1 and 2. In addition, the CNLA function circuit 404 can be configured to access the lookup table depending on the configured activation function.

[0090] In some cases, the configuration parameters may include an identification of a nonlinear activation function to be applied to the input data. Based on the determined nonlinear activation function, appropriate parameters (such as those in Table 2) may be retrieved from a memory (e.g., registers) and applied to the CNLA function circuit 404, thereby configuring it to process the input data. In some examples, a finite state machine, a memory control unit, or another controller may perform the configuration of the CNLA function circuit 404.

[0091] Notably, CNLA circuit 404 can be configured to process multiple batches of input data using the same configuration, or can update its configuration for each new batch of input data. Thus, CNLA circuit 404 provides a very flexible and efficient means for performing configurable nonlinear activations for machine learning tasks such as training and inference.

[0092] Example method for performing processing using a configurable nonlinear activation function circuit

[0093] Figure 5 An example method 500 for performing processing using a configurable non-linear activation function circuit is depicted.

[0094] Method 500 begins by determining a nonlinear activation function to apply to input data at step 502. For example, the nonlinear activation function may be one of the functions listed in Table 2, or another nonlinear activation function.

[0095] The method 500 then proceeds to step 504 to determine a parameter set for the configurable nonlinear activation function circuit based on the determined nonlinear activation function. For example, the parameters for the determined nonlinear activation function may be those in Tables 1 and 2 above.

[0096] The method 500 then proceeds to step 506, where the input data is processed using a configurable nonlinear activation function circuit based on the parameter set to generate output data. For example, the output data may be activation data for a layer of a neural network model.

[0097] In some examples, the parameter set includes a combination of one or more gain parameters, a constant parameter, and one or more approximation functions for application to input data via a configurable nonlinear activation function circuit. For example, the parameter set may be as described above with respect to Figure 1 and 2 and as discussed in Table 1 .

[0098] In some examples, method 500 further includes retrieving a parameter set from a memory based on the determined non-linear activation function. In some examples, the memory may be one or more registers storing parameter values.

[0099] In some examples, a configurable nonlinear activation function circuit includes: a first approximator configured to approximate a first function of one or more approximation functions; a second approximator configured to approximate a second function of the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value, such as a reference value. Figure 1 Depicted and described.

[0100] In some examples, the configurable nonlinear activation function circuit includes a first bypass configured to bypass the first approximator. In some examples, the configurable nonlinear activation function circuit includes a second bypass configured to bypass the second approximator. In some examples, the configurable nonlinear activation function circuit includes an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0101] In some examples, at least one of the first approximator and the second approximator is a cubic approximator. In some examples, the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator. In some examples, the other of the first approximator and the second approximator is configured to perform a minimum or maximum function, such as a function of Figure 3 In some examples, the other of the first approximator and the second approximator is configured to access a lookup table to obtain an approximation, such as Figure 3 As depicted by path 306C in FIG.

[0102] In some examples, both the first approximator and the second approximator are cubic approximators.

[0103] Notice, Figure 5 This is merely one example, and in other examples, methods such as those described herein may be implemented with more, fewer, and / or different steps.

[0104] Example CNLA architecture for softmax operation with parallel input data

[0105] Figure 6 An example architecture 600 is depicted that uses a CNLA function circuit to perform a softmax operation using parallel input data.

[0106] The Softmax (SM) function is widely used in various machine learning models, such as in many neural network (NN) architectures. For example, the SM function is often used in the last layer of a neural network (after the fully connected (FC) / dense layer) to provide multi-class classification (such as digit recognition). SM is also used for attention-based computations (e.g., in transformer models). Typically, SM maps the output of a neuron (e.g., a set of values ​​in a tensor or vector) to an interval (e.g., to a value between 0 and 1), ensuring that the sum of the mapped values ​​is 1.

[0107] As discussed above and in more detail below, exponential, logarithmic, and natural logarithmic (ln) functions can be calculated using the CNLA architecture described above, such as by using hardware lookup tables. Thus, as discussed in more detail below, CNLA circuits can be configured to provide SM functionality (also referred to as approximate SM functionality). In some aspects, if the CNLA circuits are configured to perform exponential and logarithmic operations, then these CNLA circuits are configured to provide exact SM functionality, and if the CNLA circuits are configured to perform approximate exponential and logarithmic operations (e.g., using lookup tables), then these CNLA circuits are configured to provide approximate SM functionality.

[0108] In at least one aspect, the SM function can be defined as where x is the input data (e.g., a tensor or vector containing the values ​​output by the neurons in the network), and x j is the jth element of vector x. In some aspects, the SM function can alternatively be defined as where x max =max(x j ,j=0:N-1)(for example, where x max is the maximum value in vector x). In one aspect, using the latter definition can be used to reduce the dynamic range of the data.

[0109] Additionally, in some aspects, a logarithmic softmax (Log(SM)) can be used in various architectures. In one aspect, the logarithmic softmax function can be defined as (or Where log() is the natural logarithm. In some aspects, the computationally expensive division of the SM function can be avoided by converting the domain to the logarithmic domain (e.g., by using a logarithmic softmax).

[0110] As discussed in more detail below, using the illustrated architecture 600, an approximate logarithmic softmax value 645 can be generated using a combination of CNLA blocks. In some aspects, if the original (linear) softmax domain is desired, the logarithmic softmax can be processed using another CNLA (e.g., CNLA circuit 635), as discussed in more detail below. This may generally be referred to herein as a "two-stage softmax operation."

[0111] Additionally, in some aspects, if the softmax function is used for multi-class classification, the simplified calculation can be close enough to the actual value by simply ignoring the denominator of the SM function defined above (e.g., where ). In some aspects, this may be referred to herein as a “single-stage softmax operation.” In one aspect, if a single-stage softmax operation is desired, the CNLA circuit 635 may be bypassed.

[0112] In the illustrated architecture 600, a computation array 605 provides input data (e.g., a tensor or vector) to be processed using a softmax operation (e.g., using an approximate softmax and / or approximate log-softmax operation). In the illustrated example, the computation array 605 is configured to output a set of data (e.g., multiple elements in a vector) in parallel. For example, the computation array 605 can be a CIM array. The computation array 605 can generally be used to perform any process or operation, such as to generate an output for a layer of a neural network (e.g., by multiplying the input data with a set of weights associated with the layer).

[0113] In the illustrated example, two different processing paths are shown (e.g., a first path including elements 615A, 620A, 625A, 640A, 645A, 650A, and 655A, and a second path including elements 615B, 620B, 625B, 640B, 645B, 650B, and 655B). In the illustrated example, the architecture 600 further includes several common or shared elements across the processing paths, including a maximum block 610, a summation block 630, and a CNLA circuit 635, which will be described in more detail below.

[0114] Although two paths are depicted for conceptual clarity, there may be any number of processing paths in architecture 600 (as indicated by ellipsis 612). In at least one aspect, each path is associated with a corresponding output element of computation array 605 (e.g., a corresponding value in a vector or tensor). That is, each element x in vector x j There may be corresponding paths (including corresponding CNLA circuits 625). For example, if there are sixty-four elements in the vector, then architecture 600 may include sixty-four CNLA circuits 625. In at least one aspect, each processing path corresponds to a channel of output generated by computation array 605.

[0115] As illustrated, each output element of the computation array 605 (e.g., each value in the vector x) is provided to a maximum block 610. The maximum block 610, which may be implemented using hardware or software, generally corresponds to a computation component that identifies and outputs the maximum value among its input data. Thus, in the illustrated aspect, the maximum block 610 identifies the maximum value of x in x. max (regardless of the number of values ​​in x or the number of processing paths included in architecture 600), and x max is provided to operations 615A and 615B. In one aspect, the maximum block 610 can identify the maximum value by evaluating all input values ​​in parallel. Additionally, each operation 615A and 615B receives a corresponding element from the computation array 605. That is, operation 615A may receive a first value x a, and operation 615B may receive a second value x b .

[0116] In the illustrated example, operations 615A and 615B (collectively referred to as "operation 615") are subtraction operations, where j Subtract x max Specifically, operation 615A calculates x a -x max , and operation 615B calculates x b -x max As illustrated, these values ​​are then provided to corresponding multiplexers 620A and 620B (collectively, "multiplexers 620").

[0117] In some aspects, multiplexer 620 is used to implement a two-stage softmax operation (e.g., when the desired output is a linear SM), as discussed above and described in more detail below. In one aspect, if an approximate logarithmic softmax is desired and / or if a single-stage softmax operation is being used (or during the first stage of a two-stage operation), multiplexer 620 can be used to provide the output of operation 615 directly to CNLA circuit 625. In the illustrated example, the outputs of multiplexers 615A and 615B are also provided to operations 640A and 640B, respectively, which will be discussed in more detail below.

[0118] As illustrated, the output of multiplexer 620 (or the output of operation 615 as discussed above) is then provided to a corresponding CNLA circuit 625. Specifically, the output of multiplexer 620A (e.g., based on x a is provided to the first CNLA circuit 625A, and the output of the multiplexer 620B (eg, based on x b is provided to the second CNLA circuit 625B.

[0119] In at least one aspect, CNLA circuits 625A and 625B (collectively, "CNLA circuits 625") may correspond to Figure 1 In the particular illustrated architecture 600, the CNLA circuit 625 is configured to perform an exponential operation or function. That is, the CNLA circuit 625 is configured to calculate (or approximate) an exponential output based on an input (e.g., to calculate or approximate e n , where n is the input data provided to the CNLA circuit 625).

[0120] In one aspect, to perform this exponential operation, the CNLA circuit 625 can use a gain parameter including a dependent parameter value of 0 and an independent parameter value of 1, as well as a constant value of 0, wherein the first function is bypassed and the second function is an exponential lookup table. In some aspects, the first function can be an exponential lookup table and the second function is bypassed. In this manner, the CNLA circuit 625 can provide an exponential function.

[0121] As illustrated, the output of each CNLA circuit 625 is therefore equal to or approximately equal to Specifically, the CNLA circuit 625A outputs The CNLA circuit 625B outputs In the illustrated example, the CNLA circuit 625 output from each processing path (regardless of the number of processing paths) is then provided to the summing block 630. That is, the summing block 630 can receive input from each processing path, regardless of the number of such paths. The summing block 630, which can be implemented using hardware or software, generally corresponds to a computational component that sums the received input values ​​and outputs the sum. In one aspect, the summing block 630 can sum all input values ​​in parallel. Thus, in the illustrated aspect, the summing block 630 receives input from each CNLA circuit 625. These values ​​are summed and the resulting sum is output to the CNLA circuit 635 .

[0122] In at least one aspect, the CNLA circuit 635 may correspond to Figure 1 CNLA function circuit 100. In the particularly illustrated architecture 600, CNLA circuit 635 is configured to perform a logarithmic operation or function. That is, CNLA circuit 635 is configured to compute (or approximate) a logarithm (e.g., a natural logarithm) of an input (e.g., to compute or approximate ln(n), where n is the input data provided to CNLA circuit 635).

[0123] In one aspect, to perform this exponential operation, the CNLA circuit 635 may use a gain parameter including a dependent parameter value of 0 and an independent parameter value of 1, as well as a constant value of 0, wherein the first function is bypassed and the second function is a logarithmic lookup table (e.g., a natural logarithmic lookup table). In some aspects, the first function may be a logarithmic lookup table and the second function may be bypassed. In this manner, the CNLA circuit 635 may provide a logarithmic function.

[0124] As illustrated, the output of the CNLA circuit 635 is therefore equal to or approximately equal to In the illustrated example, the output of the CNLA circuit 635 is then provided to operations 640A and 640B (collectively referred to as "operations 640"). In the illustrated example, as discussed above, each operation 640 is associated with a corresponding processing path. Specifically, operation 640A corresponds to processing the first value x.a , and operation 640B corresponds to the processing path for processing the second value x b In the illustrated example, operation 640A (or 640B) subtracts the output of CNLA circuit 635 from the output of corresponding operation 615A (or 615B).

[0125] That is, operation 640 may calculate Thus, in the illustrated example, operation 640A may calculate While operation 640B calculates

[0126] As discussed above and depicted in the illustrated example, the output of operation 640 is therefore logarithmic softmax values ​​645A and 645B (collectively referred to as "logarithmic softmax values ​​645"). That is, logarithmic softmax value 645A may be equal to or approximately log(SM(x a )), and the logarithmic softmax value 645B may be equal to or approximately log(SM(x b )). In some aspects, if the log softmax is the desired output, the log softmax value 645 may then be provided as an output from the architecture 600 (e.g., as an output from a model, or as an input to a subsequent layer of a neural network).

[0127] In at least one aspect, as discussed above, in single-stage operation, the system can provide the output of summation block 630 directly to operation 640 rather than using CNLA circuit 635. In some implementations, this can enable more efficient generation of approximate softmax values.

[0128] As discussed above, if the desired output of the architecture is a linear softmax value and the architecture uses a two-stage operation, optional paths 650A and 650B (collectively referred to as "paths 650") may be used. In the illustrated example, these paths 650 provide the generated logarithmic softmax value 645 back to the multiplexer 620 during a second or subsequent cycle or stage. That is, the depicted components may process the output of the computation array 605 in a first stage (e.g., during a first set of one or more clock cycles) to generate the logarithmic softmax value 645. During a subsequent stage (e.g., during a second set of one or more clock cycles), the logarithmic softmax value 645 may be provided back to the multiplexer 620 / CNLA circuit 625.

[0129] In some aspects, during this second phase, the multiplexers 620 may pass the logarithmic softmax values ​​645 to the corresponding CNLA circuits 625. That is, the multiplexers 620 may each provide the generated logarithmic softmax values ​​645 directly to the corresponding CNLA circuits 625. Specifically, the logarithmic softmax value 645A is provided as an input to the CNLA circuit 625A, and the logarithmic softmax value 645B is provided as an input to the CNLA circuit 625B. While the illustrated example depicts providing the logarithmic softmax values ​​645 back to the CNLA circuits 625, in some aspects, the architecture may use a second set of CNLA circuits separate from the CNLA circuits 625 to provide further exponential operations.

[0130] As illustrated and discussed above, the CNLA circuits 625 are configured to compute exponential outputs based on input data. Thus, upon receiving the logarithmic softmax value 645, each CNLA circuit 625 generates and outputs a corresponding linear softmax value 655 (e.g., linear softmax values ​​655A and 655B). That is, each CNLA circuit 625 may compute exp(log(SM(x))) for the corresponding j-th value from the input vector. j ))). In this way, the output of the CNLA circuit 625 is equal to or approximates the SM function discussed above.

[0131] In one aspect, these linear softmax values ​​655 may then be provided as output from the architecture 600 (e.g., as output from a model, or as input to subsequent layers of a neural network).

[0132] Example CNLA architecture for softmax operation with time-series input data

[0133] Figure 7 Depicted is an example architecture 700 that uses a CNLA function circuit to perform a softmax operation using sequential input data.

[0134] As discussed in more detail below, using the illustrated architecture 700, the approximate logarithmic softmax value 745 can be generated using a combination of CNLA blocks. In some aspects, if the original (linear) softmax domain is desired, the logarithmic softmax can be processed using another CNLA (e.g., CNLA circuit 735), as discussed in more detail above and below. This may generally be referred to herein as a "two-stage softmax operation."

[0135] Additionally, as discussed, if the softmax function is used for multi-class classification, the simplified calculation can be close enough to the actual value by simply ignoring the denominator of the SM function defined above (e.g., where As discussed above, this may be referred to herein as a “single-stage softmax operation.” In one aspect, if a single-stage softmax operation is desired, the CNLA circuit 735 may be bypassed.

[0136] In the illustrated architecture 700, the sequential circuit 705 provides input data (e.g., values ​​or data elements of a tensor or vector) to be processed using a softmax operation (e.g., using an approximate softmax and / or approximate log softmax operation). In the illustrated example, the sequential circuit 705 is configured to output the data elements sequentially. That is, although Figure 6 The illustrated architecture 600 is configured for parallel data input, but the illustrated architecture 700 is configured to process sequential input (e.g., where data elements are output one at a time by sequential circuit 705). For example, sequential circuit 705 may be a DMAC circuit. Sequential circuit 705 may generally be used to perform any process or operation, such as to generate the output of a layer of a neural network (e.g., by sequentially multiplying the input data elements by the corresponding weights associated with the layer).

[0137] In the example described, Figure 6 In contrast to the architecture 600 described above, a single processing path is used to process the output of the sequential circuit 705, and the architecture 700 processes the output sequentially. That is, each element x in the vector x j The illustrated processing path can be used to process sequentially. For example, each clock cycle, a new element x j may be output by sequential circuitry 705 to begin processing using the illustrated architecture 700. In at least one aspect, architecture 700 may be used to sequentially process data elements in a given channel of data output by a layer of a neural network, and a separate architecture 700 may be used to process each respective channel.

[0138] As illustrated, each output element is first provided to buffer 707. In one aspect, buffer 707 is a memory or storage component (e.g., a register file) that buffers or stores each output element from sequential circuit 705. For example, when sequential circuit 705 outputs each x jOnce all vectors x have been generated, buffer 707 can store each element until the entire vector is stored. As illustrated, buffer 707 outputs the elements to max block 710. In some embodiments, although buffer 707 is depicted, the output of sequential circuit 705 can instead be provided directly to max block 710. That is, buffer 707 can be used to provide input to operation 715, and max block 710 can also receive the buffer output or can receive input directly from sequential circuit 705 (in order).

[0139] The maximum block 727, which may be implemented using hardware or software, generally corresponds to a computational component that identifies and outputs the maximum value among its input data. Thus, in the illustrated aspect, the maximum block 710 identifies x in x. max In some embodiments, using buffer 707, max block 710 may evaluate the entire vector x at once. In some aspects that do not use buffer 707, max block 710 may evaluate the entire vector x at once upon receiving each x. j When evaluating each x sequentially j For example, the max block 707 may evaluate each newly received value x j To determine whether the value is greater than the x currently stored by the largest block 707 max If so, then the new value is available as the new / current x max Once all data elements are evaluated, the maximum block 707 may output the determined maximum value.

[0140] In the illustrated example, max block 710 and buffer 707 then output data to operation 715. That is, max block 710 outputs the maximum value in x to operation 715, which also receives the output of buffer 707 (e.g., the entire vector x). In the illustrated example, operation 715 is a subtraction operation in which x is subtracted from x (output by buffer 707). max (Output by max block 707). Specifically, operation 715 calculates x for all values ​​in x max As illustrated, these values ​​are then provided to multiplexer 720. In the illustrated example, the output of operation 715 is also provided to buffer 737.

[0141] In one aspect, in a manner similar to buffer 707, buffer 737 is a memory or storage component (e.g., a register file) that buffers or stores each output element from operation 715. For example, when operation 715 outputs each x j -x max Until all of the vector x has been generated / evaluated, the buffer 737 may store each element until the entire vector has been processed (eg, until each value x j -x max has been calculated).

[0142] In the illustrated example, multiplexer 720 is used to implement a two-stage softmax operation (e.g., when the desired output is a linear SM), as discussed above and described in more detail below. In one aspect, if an approximate logarithmic softmax is desired and / or if a single-stage softmax operation is being used (or during the first stage of a two-stage operation), multiplexer 720 can be used to provide the output of operation 715 directly to CNLA circuit 725.

[0143] In at least one aspect, the CNLA circuit 725 may correspond to Figure 1 In the particular illustrated architecture 700, the CNLA circuit 725 is configured to perform an exponential operation or function. That is, the CNLA circuit 725 is configured to compute (or approximate) an exponential output based on an input (e.g., to compute or approximate e n , where n is the input data provided to the CNLA circuit 725).

[0144] In one aspect, to perform this exponential operation, the CNLA circuit 725 can use a gain parameter including a dependent parameter value of 0 and an independent parameter value of 1, as well as a constant value of 0, wherein the first function is bypassed and the second function is an exponential lookup table. In some aspects, the first function can be an exponential lookup table and the second function is bypassed. In this manner, the CNLA circuit 725 can provide an exponential function.

[0145] As illustrated, the output of the CNLA circuit 725 is therefore equal to or approximately equal to In the illustrated example, the output of CNLA circuit 725 is then provided to summation block 730 and operation 740 .

[0146] The summation block 730, which may be implemented using hardware or software, generally corresponds to a computational component that sums the received input values ​​and outputs the sum. In one aspect, the summation block 730 may similarly sum the received input values ​​sequentially as they are received. That is, the summation block 730 may sum each newly received The value is added to the running sum. When all elements are received, the sum block 730 may output the sum.

[0147] In the illustrated example, the generated sum is output by summation block 730 to CNLA circuit 735. In at least one aspect, CNLA circuit 735 may correspond to Figure 1CNLA function circuit 100. In the particularly illustrated architecture 700, CNLA circuit 735 is configured to perform a logarithmic operation or function. That is, CNLA circuit 735 is configured to compute (or approximate) a logarithm (e.g., a natural logarithm) output based on an input (e.g., to compute or approximate ln(n), where n is input data provided to CNLA circuit 735).

[0148] In one aspect, to perform this exponential operation, the CNLA circuit 735 may use a gain parameter including a dependent parameter value of 0 and an independent parameter value of 1, as well as a constant value of 0, wherein the first function is bypassed and the second function is a logarithmic lookup table (e.g., a natural logarithmic lookup table). In some aspects, the first function may be a logarithmic lookup table and the second function may be bypassed. In this manner, the CNLA circuit 735 may provide a logarithmic function.

[0149] As illustrated, the output of the CNLA circuit 735 is therefore equal to or approximately equal to In the illustrated example, the output of the CNLA circuit 735 is then provided to operation 740. In the illustrated example, operation 740 subtracts the output of the CNLA circuit 735 from the output of the buffer 737. That is, operation 740 may calculate

[0150] As discussed above and depicted in the illustrated example, the output of operation 740 is therefore the logarithmic softmax value 745 of all x values ​​output by sequential circuit 705. In some aspects, if the logarithmic softmax is the desired output, then the logarithmic softmax value 745 may then be provided as an output from architecture 700 (e.g., as an output from a model, or as an input to a subsequent layer of a neural network).

[0151] In at least one aspect, as discussed above, in single-stage operation, the system can bypass CNLA circuit 735 and provide the output of summation block 730 directly to operation 740. In some implementations, this can enable more efficient generation of approximate softmax values.

[0152] As discussed above, if the desired output of the architecture is a linear softmax value and the architecture uses a two-stage operation, then the optional path 750 can be used. In the illustrated example, this path 750 provides the generated logarithmic softmax value 745 back to the multiplexer 720 during the second or subsequent cycle or stage. That is, the depicted components can process the output of the sequential circuit 705 in a first stage (e.g., during a first set of one or more clock cycles) to generate the logarithmic softmax value 745. During a subsequent stage (e.g., during a second set of one or more clock cycles), the logarithmic softmax value 745 can be provided back to the multiplexer 720 / CNLA circuit 725.

[0153] While the illustrated example depicts providing the logarithmic softmax value 745 back to the CNLA circuit 725 (via the multiplexer 720 ), in some aspects, the architecture may use a second CNLA circuit separate from the CNLA circuit 725 to provide further exponential operations.

[0154] As explained and discussed above, CNLA circuit 725 is configured to compute an exponential output based on input data. Thus, upon receiving logarithmic softmax value 745, CNLA circuit 725 generates and outputs a corresponding set of linear softmax values ​​755. That is, CNLA circuit 725 computes exp(x - log(SM)) for all x values ​​in the input vector. In this manner, the output of CNLA circuit 725 is equal to or approximates the SM function discussed above.

[0155] In one aspect, these linear softmax values ​​755 may then be provided as output from the architecture 700 (e.g., as output from a model, or as input to subsequent layers of a neural network).

[0156] Example method for performing an approximate Softmax operation using a CNLA function circuit

[0157] Figure 8 Depicted is an example method 800 for performing an approximate softmax operation using a configurable non-linear activation function circuit.

[0158] At block 802, an exponential output is generated by using one or more first configurable nonlinear activation function circuits (e.g., Figure 1 A first approximator 102 of Figure 2 Approximator block 206A, Figure 3 Approximator 300, Figure 6 The CNLA circuit 625 and / or Figure 7 The CNLA circuit 725) processes the input data to generate.

[0159] At block 804, the exponential outputs of one or more first configurable nonlinear activation function circuits are summed (e.g., using Figure 6 The summation block 630 and / or Figure 7 summation block 730).

[0160] At block 806, the approximate logarithmic softmax output is activated by using a second configurable nonlinear activation function circuit (e.g., Figure 1 The second approximator 104, Figure 2 Approximator block 206B, Figure 3 Approximator 300, Figure 6 The CNLA circuit 635 and / or Figure 7 The CNLA circuit 735) processes the summed exponential output to generate.

[0161] In some aspects, method 800 further includes generating an approximate softmax of the input data by processing the approximate log softmax of the input data using one or more first configurable non-linear activation function circuits.

[0162] In some aspects, the one or more first configurable non-linear activation function circuits include a plurality of first configurable non-linear activation circuits, each circuit associated with a corresponding output element from the parallelized computation array.

[0163] In some aspects, method 800 further includes determining a maximum value of output elements from the parallelized computation array, and providing the maximum value of the output elements to one or more first configurable non-linear activation function circuits.

[0164] In some aspects, the one or more first configurable non-linear activation function circuits include a single first configurable non-linear activation circuit that receives input data from the sequential computation circuit.

[0165] In some aspects, method 800 further includes determining a maximum output from the single first configurable non-linear activation circuit.

[0166] In some aspects, method 800 further includes buffering the output from the sequential computation circuit.

[0167] In some aspects, method 800 further includes: determining a nonlinear activation function to be supplied for the input data; determining a parameter set for a configurable nonlinear activation function circuit based on the determined nonlinear activation function; and processing the input data through the configurable nonlinear activation function circuit based on the parameter set to generate output data.

[0168] In some aspects, method 800 further includes retrieving a parameter set from a memory based on the determined non-linear activation function.

[0169] In some aspects, the parameter set includes a combination of one or more gain parameters, a constant parameter, and one or more approximation functions for application to input data via a configurable nonlinear activation function circuit.

[0170] In some aspects, at least one of the first configurable nonlinear activation function circuits includes: a first approximator, the first approximator being configured to approximate a first function of the one or more approximation functions; a second approximator being configured to approximate a second function of the one or more approximation functions; a first gain multiplier, the first gain multiplier being configured to multiply a first gain value based on one or more gain parameters; and a constant adder, the constant adder being configured to add a constant value based on the constant parameter.

[0171] In some aspects, at least one of the first configurable nonlinear activation function circuits further includes: a first bypass configured to bypass the first approximator; a second bypass configured to bypass the second approximator; and an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0172] In some aspects, the determined nonlinear activation function comprises an exponential function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is an exponential lookup table.

[0173] Example Processing System

[0174] Figure 9 Describes a method that can be configured to implement the present invention, such as including Figure 1-8 An example processing system 900 of the described systems, techniques, architectures, and methods.

[0175] Processing system 900 includes a central processing unit (CPU) 902, which in some examples may be a multi-core CPU. Instructions executed at CPU 902 may be loaded, for example, from a program memory associated with CPU 902 or from memory partition 924.

[0176] The processing system 900 further includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904 , a digital signal processor (DSP) 906 , a neural processing unit (NPU) 908 , a multimedia processing unit 910 , and wireless connectivity components 912 .

[0177] An NPU (such as 908) is generally a dedicated circuit configured to implement all necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. An NPU is sometimes referred to interchangeably as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0178] NPUs such as 908 can be configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs can be instantiated on a single chip such as a system on a chip (SoC), while in other examples, multiple NPUs can be part of a dedicated machine learning accelerator device.

[0179] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, the two tasks can generally still be performed independently.

[0180] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (typically labeled or tagged), iterating on the dataset, and then adjusting model parameters (such as weights and biases) to improve model performance. Generally speaking, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0181] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs can thus be configured to input new data segments and quickly process them through an already trained model to generate model output (e.g., inference).

[0182] In some aspects, the NPU 908 may be implemented as part of one or more of the CPU 902 , GPU 904 , and / or DSP 906 .

[0183] In some aspects, the wireless connectivity component 912 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 912 is further connected to one or more antennas 914.

[0184] The processing system 900 may also include one or more sensor processing units 916 associated with any manner of sensors, one or more image signal processors (ISPs) 918 associated with any manner of image sensors, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0185] The processing system 900 may also include one or more input and / or output devices 922, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0186] In some examples, one or more processors of processing system 900 may be based on the ARM or RISC-V instruction set.

[0187] Processing system 900 also includes various circuits in accordance with various aspects described herein.

[0188] In this example, the processing system 900 includes a compute-in-memory (CIM) circuit 926 that can be configured to perform an efficient multiply and accumulate (MAC) function for processing machine learning model data. The processing system 900 further includes a configurable nonlinear activation (CNLA) function circuit 928. In some cases, the CNLA function circuit 928 can be similar to the one described in reference to FIG. Figure 2 The CNLA function circuit 200 is described. The CNLA function circuit 928 and other circuits not depicted may be configured to perform various aspects of the methods described herein, such as those related to Figure 4 Process 400.

[0189] In some examples, CNLA function circuit 928 may be implemented as part of another processing unit, such as CPU 902 , GPU 904 , DSP 906 , or NPU 908 .

[0190] The processing system 900 further includes a memory 924 , which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 924 includes computer-executable components that can be executed by one or more of the aforementioned components of the processing system 900 .

[0191] Specifically, in this example, memory 924 includes a determination component 924A, a configuration component 924B, a processing component 924C, a retrieval component 924D, non-linear activation function parameters 924E, (several) lookup tables 924F, and model parameters 924G (e.g., weights, biases, and other machine learning model parameters). One or more of the depicted components and other components not depicted can be configured to perform various aspects of the methods described herein.

[0192] In general, the processing system 900 and / or its components may be configured to perform the methods described herein.

[0193] It is noted that in other aspects, aspects of the processing system 900 may be omitted, such as where the processing system 900 is a server computer, etc. For example, in other aspects, the multimedia component 910, wireless connectivity 912, sensors 916, ISP 918, and / or navigation component 920 may be omitted. Furthermore, aspects of the processing system 900 may be distributed.

[0194] Notice, Figure 9 This is merely one example, and in other examples, alternative processing systems having more, fewer, and / or different components may be used.

[0195] Sample Clauses

[0196] Implementation examples are described in the following numbered clauses:

[0197] Item 1: A processor comprising: one or more first configurable nonlinear activation function circuits, the one or more first configurable nonlinear activation function circuits being configured to perform an exponential function on input data; a summation circuit, the summation circuit being configured to receive output data of the one or more first configurable nonlinear activation function circuits; and a second configurable nonlinear activation function circuit, the second configurable nonlinear activation function circuit being configured to receive the output data of the summation circuit, perform a natural logarithm function, and output an approximate logarithm softmax of the input data.

[0198] Item 2: A processor as in Item 1, wherein: the second configurable nonlinear activation function circuit is configured to output an approximate logarithmic softmax of the input data during a first loop, wherein during a second loop following the first loop, the approximate logarithmic softmax of the input data is provided as input to one or more first configurable nonlinear activation function circuits, and wherein the one or more first configurable nonlinear activation function circuits are configured to output an approximate softmax of the input data based on the approximate logarithmic softmax of the input data.

[0199] Clause 3: The processor of any of clauses 1-2, wherein the one or more first configurable non-linear activation function circuits comprises a plurality of first configurable non-linear activation circuits, each circuit being associated with a corresponding output element from the parallelized computation array.

[0200] Clause 4: The processor of any of clauses 1-3, further comprising a maximum circuit configured to: receive an output element from the parallelized computation array as input, and output a maximum value from the output element to one or more first configurable nonlinear activation function circuits.

[0201] Clause 5: The processor of any of clauses 1-4, wherein the one or more first configurable non-linear activation function circuits include a single first configurable non-linear activation circuit configured to receive input data from the sequential computation circuit.

[0202] Clause 6: The processor of any of clauses 1-5, further comprising a maximum circuit configured to: receive as input an output from the single first configurable non-linear activation circuit, and output a maximum value from the sequential computation circuit.

[0203] Clause 7: The processor of any of clauses 1-6, further comprising a memory buffer configured to buffer output from the sequential computation circuit.

[0204] Clause 8: A processor as in any of clauses 1-7, wherein at least one of the first configurable nonlinear activation function circuits is configured to: determine a nonlinear activation function to supply for input data; determine a parameter set for the nonlinear activation function based on the determined nonlinear activation function; and generate output data based on application of the parameter set for the nonlinear activation function.

[0205] Clause 9: A processor as in any of clauses 1-8, wherein at least one of the one or more first configurable nonlinear activation function circuits comprises: a first approximator configured to approximate the first function using one or more first function parameters in a parameter set; a second approximator configured to approximate the second function using one or more second function parameters in the parameter set; a gain multiplier configured to multiply a gain value based on the one or more gain parameters in the parameter set; and a constant adder configured to add a constant value based on the constant parameter in the parameter set.

[0206] Clause 10: The processor of any of clauses 1-9, wherein both the first approximator and the second approximator are cubic approximators.

[0207] Clause 11: The processor of any of clauses 1-10, wherein one of the first approximator or the second approximator is a cubic approximator.

[0208] Clause 12: The processor of any of clauses 1-11, wherein the other of the first approximator or the second approximator is a quadratic approximator or a linear approximator.

[0209] Clause 13: The processor of any of clauses 1-12, wherein the other of the first approximator or the second approximator is configured to access a lookup table to obtain the approximation.

[0210] Clause 14: The processor of any of clauses 1-13, wherein the other of the first approximator or the second approximator is configured to perform a minimum or maximum function.

[0211] Clause 15: The processor of any of clauses 1-14, wherein the determined nonlinear activation function comprises an exponential function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is an exponential lookup table.

[0212] Clause 16: A processor as in any of clauses 1-15, wherein the second configurable nonlinear activation function circuit comprises: a first approximator configured to approximate the first function using one or more first function parameters in a parameter set; a second approximator configured to approximate the second function using one or more second function parameters in a parameter set; a gain multiplier configured to multiply a gain value based on one or more gain parameters in the parameter set; and a constant adder configured to add a constant value based on a constant parameter in the parameter set, the gain parameter comprising a dependent parameter value of 0 and an independent parameter value of 1, the constant value being 0, the first function being bypassed, and the second function being a natural logarithm lookup table.

[0213] Item 17: A method for processing input data through a set of configurable nonlinear activation function circuits, comprising: generating an exponential output by processing the input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function; summing the exponential outputs of the one or more first configurable nonlinear activation function circuits; and generating an approximate logarithmic softmax output by processing the summed exponential output using a second configurable nonlinear activation function circuit configured to perform a natural logarithm function.

[0214] Clause 18: The method of clause 17, further comprising: generating an approximate softmax of the input data by processing the approximate logarithmic softmax of the input data using one or more first configurable nonlinear activation function circuits.

[0215] Clause 19: The method of any of clauses 17-18, wherein the one or more first configurable non-linear activation function circuits comprises a plurality of first configurable non-linear activation circuits, each circuit being associated with a corresponding output element from the parallelized computation array.

[0216] Clause 20: The method of any of clauses 17-19, further comprising: determining a maximum value of output elements from the parallelized computation array, and providing the maximum value from the output elements to one or more first configurable nonlinear activation function circuits.

[0217] Clause 21: The method of any of clauses 17-20, wherein the one or more first configurable non-linear activation function circuits include a single first configurable non-linear activation circuit that receives input data from the sequential computation circuit.

[0218] Clause 22: The method of any of clauses 17-21, further comprising: determining a maximum output from the single first configurable non-linear activation circuit.

[0219] Clause 23: The method of any of clauses 17-22, further comprising: buffering an output from the sequential computation circuit.

[0220] Clause 24: A method as in any of clauses 17-23, further comprising: determining a nonlinear activation function to be supplied for input data; determining a parameter set for a configurable nonlinear activation function circuit based on the determined nonlinear activation function; and processing the input data through the configurable nonlinear activation function circuit based on the parameter set to generate output data.

[0221] Clause 25: The method of any of clauses 17-24, further comprising: retrieving a parameter set from a memory based on the determined non-linear activation function.

[0222] Clause 26: The method of any of clauses 17-25, wherein the parameter set comprises a combination of one or more gain parameters, a constant parameter, and one or more approximation functions for application to the input data via a configurable non-linear activation function circuit.

[0223] Clause 27: A method as in any of clauses 17-26, wherein at least one of the first configurable nonlinear activation function circuits comprises: a first approximator configured to approximate a first function of the one or more approximation functions; a second approximator configured to approximate a second function of the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value based on the constant parameter.

[0224] Clause 28: The method of any of clauses 17-27, wherein at least one of the first configurable nonlinear activation function circuits further comprises: a first bypass configured to bypass the first approximator; a second bypass configured to bypass the second approximator; and an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0225] Clause 29: The method of any of clauses 17-28, wherein: the determined nonlinear activation function comprises an exponential function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is an exponential lookup table.

[0226] Clause 30: A method as in any of clauses 17-29, wherein: the second configurable nonlinear activation function circuit includes: a first approximator, the first approximator is configured to approximate the first function using one or more first function parameters in a parameter set; a second approximator, the second approximator is configured to approximate the second function using one or more second function parameters in the parameter set; a gain multiplier, the gain multiplier is configured to multiply a gain value based on one or more gain parameters in the parameter set; and a constant adder, the constant adder is configured to add a constant value based on a constant parameter in the parameter set, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is a natural logarithm lookup table.

[0227] Clause 31: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform the method according to any of clauses 17-30.

[0228] Clause 32: A processing system comprising means for performing the method according to any of clauses 17-30.

[0229] Clause 33: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any of clauses 17-30.

[0230] Clause 34: A computer program product embodied on a computer-readable storage medium, the computer-readable storage medium comprising code for performing the method according to any of clauses 17-30.

[0231] Additional considerations

[0232] The foregoing description is provided to enable anyone skilled in the art to practice the various aspects described herein. The examples discussed herein are not intended to limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made to the functions and arrangements of the elements discussed without departing from the scope of this disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Moreover, features described with reference to some examples may be combined in some other examples. For example, a device may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope of this disclosure is intended to cover such devices or methods practiced using other structures, functionalities, or structures and functionalities that are in addition to or different from the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be implemented by one or more elements of the claims.

[0233] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0234] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass: a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0235] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or another data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0236] As used herein, in the context of sharing electronic signals and data between the various elements described herein, the term "connected to" may generally refer to data communication between corresponding elements that are connected to each other. In some cases, the elements may be directly connected to each other, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between the corresponding elements that are directly connected to each other. In other cases, the elements may be indirectly connected to each other, such as via one or more data buses or similar shared circuit systems and / or integrated circuit elements for conveying signals and data between the corresponding elements that are indirectly connected to each other.

[0237] Each method disclosed herein includes one or more steps or actions for implementing the method. These method steps and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of the specific steps and / or actions can be changed without departing from the scope of the claims. In addition, the various operations of the above-mentioned methods can be performed by any suitable device capable of performing the corresponding functions. These devices may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the accompanying drawings, these operations may have corresponding paired devices plus functional components with similar numbers.

[0238] The following claims are not intended to be limited to the aspects shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean "there is and only one" (unless specifically stated as such), but rather "one or more". Unless specifically stated otherwise, the term "some / a certain" refers to one or more. No element of a claim should be interpreted under the provisions of 35 U.S.C. § 112 (f) unless the element is explicitly described using the phrase "device for..." or in the case of a method claim, the element is described using the phrase "step for...". The elements of the various aspects described throughout this disclosure are expressly incorporated herein by reference for all structural and functional equivalents currently or hereafter known to those of ordinary skill in the art, and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly stated in the claims.

Claims

1. A processor, comprising: one or more first configurable non-linear activation function circuits, the one or more first configurable non-linear activation function circuits configured to perform an exponential function on input data; a summing circuit configured to receive output data of the one or more first configurable nonlinear activation function circuits; as well as A second configurable nonlinear activation function circuit is configured to receive the output data of the summation circuit, perform a natural logarithm function, and output an approximate logarithm softmax of the input data.

2. The processor of claim 1 , wherein the second configurable nonlinear activation function circuit is configured to output the approximate logarithmic softmax of the input data during a first loop, wherein during a second loop following the first loop, the approximate logarithmic softmax of the input data is provided as input to the one or more first configurable nonlinear activation function circuits, and wherein the one or more first configurable nonlinear activation function circuits are configured to output the approximate softmax of the input data based on the approximate logarithmic softmax of the input data.

3. The processor of claim 1 , wherein the one or more first configurable nonlinear activation function circuits comprise a plurality of first configurable nonlinear activation circuits, each circuit being associated with a corresponding output element from a parallelized computation array.

4. The processor of claim 3 , further comprising a maximum circuit configured to: receiving as input an output element from the parallelized computation array; and The maximum value from the output elements is output to the one or more first configurable non-linear activation function circuits.

5. The processor of claim 1 , wherein the one or more first configurable nonlinear activation function circuits include a single first configurable nonlinear activation circuit, the single first configurable nonlinear activation circuit configured to receive the input data from a sequential computation circuit.

6. The processor of claim 5 , further comprising a maximum circuit configured to: receiving as input an output from the single first configurable non-linear activation circuit; and Outputs the maximum value from the sequential calculation circuit.

7. The processor of claim 5, further comprising a memory buffer configured to buffer output from the sequential computation circuit.

8. The processor of claim 1 , wherein at least one of the first configurable non-linear activation function circuits is configured to: determining a nonlinear activation function to be applied to the input data; determining a parameter set for the nonlinear activation function based on the determined nonlinear activation function; and Output data is generated based on application of the set of parameters for the non-linear activation function.

9. The processor of claim 8, wherein at least one of the one or more first configurable non-linear activation function circuits comprises: a first approximator configured to approximate a first function using one or more first function parameters in the parameter set; a second approximator configured to approximate a second function using one or more second function parameters in the parameter set; a gain multiplier configured to multiply a gain value based on one or more gain parameters in the parameter set; as well as A constant adder is configured to add a constant value based on a constant parameter in the parameter set.

10. The processor of claim 9, wherein both the first approximator and the second approximator are cubic approximators.

11. The processor of claim 9, wherein one of the first approximator or the second approximator is a cubic approximator.

12. The processor of claim 11, wherein the other of the first approximator or the second approximator is a quadratic approximator or a linear approximator.

13. The processor of claim 11, wherein the other of the first approximator or the second approximator is configured to access a lookup table to obtain an approximation.

14. The processor of claim 11, wherein the other of the first approximator or the second approximator is configured to perform a minimum or maximum function.

15. The processor of claim 9, wherein: The nonlinear activation functions determined include exponential functions, The gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, The constant value is 0, The first function is bypassed, and The second function is an exponential lookup table.

16. The processor of claim 1 , wherein: The second configurable nonlinear activation function circuit includes: a first approximator configured to approximate a first function using one or more first function parameters in the parameter set; a second approximator configured to approximate a second function using one or more second function parameters in the parameter set; a gain multiplier configured to multiply a gain value based on one or more gain parameters in the parameter set; and a constant adder configured to add a constant value based on a constant parameter in the parameter set, The gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, The constant value is 0, The first function is bypassed, and The second function is a natural logarithm lookup table.

17. A method for processing input data through a set of configurable nonlinear activation function circuits, comprising: generating an exponential output by processing input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function; summing the exponential outputs of the one or more first configurable nonlinear activation function circuits; as well as An approximate logarithmic softmax output is generated by processing the summed exponential output using a second configurable non-linear activation function circuit configured to perform a natural logarithm function.

18. The method of claim 17, further comprising generating an approximate softmax of the input data by processing the approximate log softmax of the input data using the one or more first configurable nonlinear activation function circuits.

19. The method of claim 17, wherein the one or more first configurable nonlinear activation function circuits comprises a plurality of first configurable nonlinear activation circuits, each circuit being associated with a corresponding output element from a parallelized computation array.

20. The method of claim 19, further comprising: determining a maximum value of the output elements from the parallelized computation array; as well as The maximum value from the output elements is provided to the one or more first configurable non-linear activation function circuits.

21. The method of claim 17, wherein the one or more first configurable non-linear activation function circuits include a single first configurable non-linear activation circuit that receives input data from a sequential computation circuit.

22. The method of claim 21, further comprising determining a maximum output from the single first configurable non-linear activation circuit.

23. The method of claim 21, further comprising buffering an output from the sequential computation circuit.

24. The method of claim 17, further comprising: Determine the nonlinear activation function to be applied to the input data; determining a parameter set for a configurable nonlinear activation function circuit based on the determined nonlinear activation function; as well as Input data is processed by the configurable nonlinear activation function circuit based on the parameter set to generate output data.

25. The method of claim 24, further comprising retrieving the parameter set from a memory based on the determined non-linear activation function.

26. The method of claim 24, wherein the parameter set comprises a combination of one or more gain parameters, a constant parameter, and one or more approximation functions for application to the input data via the configurable nonlinear activation function circuit.

27. The method of claim 26, wherein at least one of the first configurable nonlinear activation function circuits comprises: a first approximator configured to approximate a first function of the one or more approximation functions; a second approximator configured to approximate a second function of the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on the one or more gain parameters; as well as A constant adder is configured to add a constant value based on the constant parameter.

28. The method of claim 27, wherein at least one of the first configurable nonlinear activation function circuits further comprises: a first bypass configured to bypass the first approximator; a second bypass configured to bypass the second approximator; as well as An input data bypass is configured to bypass the first approximator and provide the input data to the second approximator.

29. The method of claim 28, wherein: The nonlinear activation functions determined include exponential functions, The gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, The constant value is 0, The first function is bypassed, and The second function is an exponential lookup table.

30. The method of claim 17, wherein: The second configurable nonlinear activation function circuit includes: a first approximator configured to approximate a first function using one or more first function parameters in the parameter set; a second approximator configured to approximate a second function using one or more second function parameters in the parameter set; a gain multiplier configured to multiply a gain value based on one or more gain parameters in the parameter set; and a constant adder configured to add a constant value based on a constant parameter in the parameter set, The gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, The constant value is 0, The first function is bypassed and the second function is a natural logarithm lookup table.

Images