Configurable nonlinear activation function circuits

Abstract

Some aspects of the present disclosure provide a processor comprising a configurable nonlinear activation function circuit, the configurable nonlinear activation function circuit configured to determine a set of parameters for a nonlinear activation function based on a selected nonlinear activation function, and to generate output data based on application of the set of parameters for the nonlinear activation function, where the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two continuous linear approximators, each linear approximator of the at least two continuous linear approximators configured to approximate a linear function using one or more function parameters of the set of parameters.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Application No. 17 / 807,125, filed June 15, 2022, which is a continuation-in-part of U.S. Application No. 17 / 467,079, filed September 3, 2021, the entire contents of both of which are incorporated herein by reference.

[0002] Aspects of the present disclosure relate to processing nonlinear activation functions for machine learning models, and in particular to a configurable nonlinear activation function circuit. [Background technology]

[0003] Introduction

[0003] Machine learning is generally the process of creating a trained model (e.g., an artificial neural network) that represents a generalized fit to a set of training data. Applying the trained model to new data allows for the creation of inferences, which can be used to gain insight into the new data.

[0004]

[0004] As the use of machine learning has proliferated to enable various machine learning (or artificial intelligence) tasks, a need has arisen for more efficient processing of machine learning model data. In some cases, dedicated hardware such as machine learning (or artificial intelligence) accelerators or processors or similar circuits may be used to improve the capacity of a processing system to process machine learning model data. For example, processing data with a nonlinear activation function may be distributed to processors other than a first-order matrix multiplication processor. However, distributing various aspects of processing a machine learning model across different processing devices may incur latency, memory usage, power usage, and other processing penalties.

[0005]

[0005] Therefore, there is a need for improved techniques for processing machine learning model data using non-linear activation functions. Summary of the Invention

[0006]

[0006] Some aspects provide a processor having a configurable nonlinear activation function circuit, the configurable nonlinear activation function circuit configured to determine a nonlinear activation function for application to input data, determine a set of parameters for the nonlinear activation function based on the determined nonlinear activation function, and generate output data based on application of the set of parameters for the nonlinear activation function.

[0007]

[0007] A further aspect provides a method for processing input data by a configurable nonlinear activation function circuit, the method comprising determining a nonlinear activation function for application to the input data, determining a set of parameters for the configurable nonlinear activation function circuit based on the determined nonlinear activation function, and processing the input data using the configurable nonlinear activation function circuit based on the set of parameters to generate output data.

[0008]

[0008] A further aspect provides a processor comprising a configurable nonlinear activation function circuit, the configurable nonlinear activation function circuit configured to determine a set of parameters for a nonlinear activation function based on a selected nonlinear activation function, and to generate output data based on application of the set of parameters for the nonlinear activation function, wherein the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two successive linear approximators, each linear approximator of the at least two successive linear approximators configured to approximate a linear function using one or more function parameters from the set of parameters.

[0009]

[0009] A further aspect provides a method for processing data using a configurable nonlinear activation function circuit comprising determining a set of parameters for the nonlinear activation function based on a selected nonlinear activation function, and generating output data based on application of the set of parameters for the nonlinear activation function, wherein the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two continuous linear approximators, each linear approximator of the at least two continuous linear approximators configured to approximate a linear function using one or more function parameters from the set of parameters.

[0010]

[0010] Other aspects provide a processing system configured to perform the above-mentioned methods as well as methods described herein; a non-transitory computer-readable medium comprising instructions which, when executed by one or more processors of the processing system, cause the processing system to perform the above-mentioned methods as well as methods described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the above-mentioned methods as well as methods further described herein; and a processing system comprising means for performing the above-mentioned methods as well as methods further described herein.

[0011] The following description and the annexed drawings set forth in detail certain illustrative features of the one or more embodiments.

[0012]

[0012] The accompanying drawings illustrate several aspects of the one or more embodiments and therefore should not be considered as limiting the scope of the present disclosure. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an example configurable nonlinear activation (CNLA) function circuit. [Diagram 2]

[0014] 2 illustrates an example circuit block for implementing a bypassable approximator block as described with respect to FIG. 1. [Diagram 3]

[0015] FIG. 1 illustrates an example approximator. [Figure 4]

[0016] FIG. 1 illustrates an example machine learning model process flow. [Diagram 5]

[0017] FIG. 1 illustrates an exemplary method for performing processing using a configurable nonlinear activation function circuit. [Figure 6]

[0018] FIG. 1 illustrates an example of a pipelined successive-linear-approximation architecture that may be used to implement non-linear functions used in machine learning model acceleration. [Figure 7]

[0019] FIG. 1 illustrates an example of a tanh nonlinear activation function with non-uniform segments for defining the approximated function output. [Figure 8]

[0020] FIG. 2 illustrates an example of a circuit for implementing a pipelined continuous linear approximation architecture. [Figure 9]

[0021] 9 illustrates an example of a linear approximator circuit, which may be an example of a linear approximator element in the circuit of FIG. 8. [Figure 10]

[0022] FIG. 1 illustrates an example processing system that may be configured to perform the methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0023] For ease of understanding, wherever possible, like reference numbers have been used to designate like elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.

[0015]

[0024] Aspects of the present disclosure provide improved techniques for processing non-linear activation functions associated with machine learning models.

[0016]

[0025] Nonlinear activation is an important component of various types of machine learning models, including neural network models. Some nonlinear activation functions are implemented as piecewise linear functions (e.g., rectified linear unit (ReLU), leaky ReLU, etc.), while other nonlinear activation functions require complex mathematical functions (e.g., sigmoid, hyperbolic tangent (tanh), etc.). In some cases, the complex mathematical functions may be implemented using interpolation, such as cubic spline interpolation. For example, the interpolated output value may be determined in some aspects using a lookup table (LUT) to match the output value with the input value. When a target input value is not mapped in the LUT, LUT values ​​associated with input values ​​adjacent to the target input value may be used to interpolate an output value for the target input value.

[0017]

[0026] Conventionally, nonlinear activation functions may be implemented in software rather than hardware due to the wide range of possible activation functions available in machine learning models. However, such implementations generally require moving model data between processing devices (e.g., between a neural processing unit (NPU) performing matrix multiplication and accumulation and a digital signal processor (DSP) processing the nonlinear activation function), thus incurring power and latency penalties. When nonlinear activation functions are implemented in hardware, they are generally limited to supporting only a small number of nonlinear activation functions, and therefore cannot be configured to support evolving machine learning model architectures without falling back to outsourcing the nonlinear activation function processing to distributed processing units.

[0018]

[0027] For example, the rectified linear unit (ReLU) is a commonly used activation function in deep learning models. The function returns 0 if it receives a negative input, and returns the input x otherwise. It can therefore be written as f(x)=max(0,x). The ReLU function is generally not implemented by a first-order matrix multiply-accumulate processing unit, such as a compute-in-memory (CIM) array, in some examples. Thus, the need to distribute a ReLU function, or another nonlinear activation function, is costly from a processing standpoint. Moreover, as the activation function becomes more complex, the processing costs become larger as well (e.g., to implement the relatively higher power exponential and division operations that are part of some nonlinear activation functions, as described further below).

[0019]

[0028] To overcome the shortcomings of conventional solutions, aspects described herein relate to a configurable nonlinear activation (CNLA) function circuit that can be implemented in hardware for efficient processing. In particular, because the CNLA function circuit can be implemented in hardware, the CNLA function can be co-located with other processing circuits that are optimized for other machine learning model processing tasks, such as CIM arrays and digital multiply-and-accumulate (DMAC) circuits that are optimized to perform vector and matrix multiply-accumulate functions.

[0020]

[0029] To improve processing efficiency, aspects described herein may use polynomial approximations to approximate complex functions, such as those that may be used in nonlinear activation functions. In some cases, aspects described herein may use series expansions, such as Taylor series. In general, a Taylor series of a function (e.g., f(x)) is an infinite sum of terms expressed in terms of the derivative of the function at a single point. For many functions, the function and the sum of its Taylor series are equal near this point. The partial sum formed by the first n+1 terms of the Taylor series is a polynomial of degree n, called the nth Taylor polynomial of the function. Taylor polynomials therefore allow for processing-efficient approximations of functions, which generally get better as n increases.

[0021]

[0030] The CNLA function circuit described herein may implement one or more polynomial approximation blocks, such as a cubic approximation block, which generally improves cubic spline interpolation to make it more efficient and more generalized to cover a wider variety of nonlinear activation functions. Moreover, the CNLA function circuit may be implemented as a pipelined digital block that can use nonlinearly segmented lookup tables (LUTs) and mixed-order approximations (e.g., pipelined linear, quadratic, and cubic approximations). Thus, unlike conventional nonlinear activation function circuits, the CNLA function circuit described herein may be configured to meet many different performance goals.

[0022]

[0031] Thus, the CNLA function circuit described herein provides a technical solution to the technical problem of implementing a wide range of nonlinear activation functions in a machine learning model processing system. Furthermore, the CNLA function circuit described herein provides a technical improvement by increasing model processing performance compared to existing solutions, including lower latency, lower power usage, improved memory efficiency, etc., as described herein.

[0023] Exemplary Configurable Nonlinear Activation Function Circuit

[0032] FIG. 1 illustrates an exemplary configurable non-linear activation (CNLA) function circuit 100.

[0024]

[0033] In general, the CNLA function circuit 100 may be configured to receive input data 101 (e.g., output values ​​from a layer of a machine learning model) and perform various non-linear activation functions to generate (e.g., "activate") output data 114. The CNLA function circuit 100 may be co-located and pipelined with other machine learning model processing circuits, such as CIM arrays, DMACs, etc., and may be configured to perform activation functions based on the output of the other machine learning model processing circuits.

[0025]

[0034] In some examples, the input data 101 may be received from a buffer or other memory, in other examples, the input data 101 may be received directly from the output of another processing block, such as the output of a CIM array or another vector and matrix multiply-accumulate block.

[0026]

[0035] The CNLA function circuit 100 includes a first approximator block 102, which may be generally configured to perform a hardware-based mathematical function on input data, such as input data 101. An exemplary approximator is described in detail with respect to FIG.

[0027]

[0036] In some cases, the first approximator may be a linear approximator (e.g., configured to implement a function such as ax+b), 2 +bx+c) or a quadratic approximator (e.g., ax 3 +bx 2The first approximator 102 may be one of a cubic approximator (configured to implement a function such as x+cx+d, where x is the input data and a, b, c, and d are configurable parameters). In general, a linear, quadratic, or cubic approximator may be used to approximate any given function, which may or may not be a polynomial function. The first approximator 102 may be configured with parameters, e.g., retrieved from a memory, a register, a lookup table, etc. As described in more detail below with respect to Table 2, these different forms of approximation and associated configurable parameters may be used to approximate many types of non-linear activation functions.

[0028]

[0037] The CNLA function circuit 100 further includes a second approximator block 104, which, like the first approximator block 102, may be configured to implement hardware-based mathematical functions, such as generally linear, quadratic, or cubic functions. As described in more detail below, the CNLA function circuit 100 may be configured to use the first approximator block 102 and the second approximator block 104 consecutively for more complex functions, such that the output of the first approximator block 102 becomes the input to the second approximator block 104. The CNLA function circuit 100 may be further configured to use only one of the first approximator block 102 or the second approximator block 104 when a simpler nonlinear function is being processed, thereby conserving power.

[0029]

[0038] In some implementations, the first approximator 102 and the second approximator 104 may comprise the same circuit block (e.g., two instances of the same circuit element in the circuit 100). For example, the first approximator 102 and the second approximator 104 may comprise, in some aspects, a third-order approximator. In other implementations, the first approximator 102 and the second approximator 104 may comprise different circuit elements, in which case, generally, the second approximator 104 will comprise a third-order approximator and the first approximator 102 will comprise a lower-order approximator, such as a quadratic or linear approximator. However, in other embodiments, the orders of the higher-order and lower-order approximators may be reversed.

[0030]

[0039] The CNLA function circuit 100 includes a configurable bypass 105, which allows the first approximator 102 to be bypassed in various scenarios, such as when the function requires only a lower order approximator than the first approximator 102 and the second approximator 104 is such a lower order approximator. For example, when the first approximator 102 is bypassed via the configurable bypass 105, the input data 101 is instead provided directly to the second approximator 104 and is not processed by the first approximator 102. In various aspects, the first approximator 102 may be a higher order approximator compared to the second approximator 104, or vice versa, or they may be of the same order (e.g., both linear, quadratic, or cubic). The configurable bypass 105 allows for saving processing time and energy when only one approximator is required.

[0031]

[0040] The CNLA function circuit 100 further includes another configurable bypass 107, which allows the second approximator 104 to be bypassed in various scenarios, such as when the function requires only a first approximation, where the first approximator 102 can be implemented without the second approximator 104. For example, when the second approximator 104 is bypassed via the configurable bypass 107, the output of the first approximator 102 is provided directly to the multiplier 108.

[0032]

[0041] In general, the configurable bypasses 105 and 107 allow the CNLA function circuit 100 to be configured for maximum versatility in various scenarios while conserving power and avoiding unnecessary circuit block processing. Additionally, the configurable bypasses allow asymmetric and antisymmetric nonlinear activation functions to be configured for processing by the CNLA function circuit 100. Figure 2 shows an exemplary circuit embodiment for implementing the configurable bypasses 105 and 107 (e.g., bypasses 205A and 205B).

[0033]

[0042] The CNLA function circuit 100 further includes a gain block 106 configured to provide a gain value to the multiplier 108. In some aspects, the gain block 106 is configured to generate a gain value 109 based on a gain function implemented by the gain block 106. In one example, the gain function may be of the form g=ax+b, where g is the gain value, x is the input data 101 value, and a and b are configurable parameters. More generally, the gain block 106 may multiplicatively (a) and / or additively (b) modify the input data to generate the gain value.

[0034]

[0043] The gain value 109 generated by the gain block 106 is multiplied by the output of the first approximator 102 and / or the second approximator 104 via a multiplier 108. In other aspects, the gain block 106 may be configured to generate gain values ​​that are not based on a function of the input data 101 (e.g., by setting a to 0 in the above equation for g). In general, the parameters (e.g., a and b in the above example) or values ​​for the gain block 106 may be retrieved, for example, from a memory, a register, a look-up table, or the like.

[0035]

[0044] The CNLA function circuit 100 further includes a constant block 110 configured to store a configurable (e.g., programmable) constant value 113, and an adder 112 configured to add the constant value 113 to the output of the multiplier 108 (e.g., a gain multiplier). The constant value 113 stored in the constant block 110 may be retrieved, for example, from a memory, a register, a look-up table, etc.

[0036]

[0045] The inclusion and arrangement of the first approximator block 102, the second approximator block 104, the configurable bypasses 105 and 107, the gain block 106, the multiplier 108, the constant block 110, and the adder 112 allow the CNLA function circuit 100 to be configured to implement a wide variety of known and later developed nonlinear activation functions. Moreover, the CNLA function circuit 100 can be efficiently configured to handle a wide variety of nonlinear activation functions by simply updating the parameters for the first approximator 102, the second approximator 104, the gain block 106, and the constant block 110. When both the approximator block 102 and the approximator block 104 are used to simulate a nonlinear function, each may be referred to as implementing an individual function (e.g., a first function for the first approximator block 102 and a second function for the second approximator 104). This design beneficially supports any asymmetric nonlinear curve for a complex function.

[0037]

[0046] Table 1 below provides example parameters for various nonlinear activation functions that the CNLA function circuit 100 of FIG. 1 may be configured to implement, including parameters for the approximator blocks 206A and 206B of FIG. 2. Note that in Table 1, the gain is considered to have the form ax+b, as in the example of the gain block 106 in FIG. 1, but in other embodiments the gain may be a scalar value, or a different functional form. Similarly, a quadratic approximator may be of the form ax 2 +bx+c, and the third-order approximator is of the form ax 3 +bx 2+cx+d. In the tables below, subscripts are used to indicate parameter assignments, e.g., G for the gain parameter, 1 for the first approximator, and 2 for the second approximator parameter.

[0038] [Table 1-1] [Table 1-2] [Table 1-3]

[0039]

[0047] Note that in the above ELU function, the α parameter can be configured as a hyperparameter by the model designer.

[0040]

[0048] In particular, in some implementations, the parameters for the approximator may be given in a form (e.g., cubic with a, b, c, and d parameters, or quadratic with a, b, and c parameters) even if the approximator implements a lower-order function (e.g., linear). This is because, for example, setting the cubic parameter a to 0 effectively reduces the approximation equation to a lower-order quadratic function, and similarly, setting the quadratic parameter a to 0 effectively reduces the approximation equation to a linear form. Thus, the approximator may be configured for a "quadratic function," for example, when it is configured with quadratic parameters, but the resulting parameters may reduce the function to a linear function, as in the case of the ReLU example in Table 2 above. This may allow the normalization of the parameter set to be configured by the parameters regardless of the order of the underlying function, thereby simplifying the implementation.

[0041]

[0049] 2 illustrates example circuit blocks 202 and 204 for implementing bypassable approximator blocks 206A and 206B, which in one example may correspond to the first approximator block 102 and the second approximator block 104 of FIG.

[0042]

[0050] 2, circuit block 202 is configured to control the use of function block 214A, which in this example includes a first approximator 206A and a minimum and maximum function block 208A. Similarly, circuit block 204 controls the use of function block 214B, which in this example includes a minimum and maximum function block 208B and a second approximator 206B. The first approximator block 206A and the second approximator block 206B may be configured to implement a non-linear activation function, such as those described above with respect to Table 1.

[0043]

[0051] Note that while the first approximator 102 in FIG. 1 requires only one input, the circuit block 202 includes two input ports 201A and 201B, allowing for multiple inputs. The illustrated configuration of the circuit block 202 may be adopted to present the same external interface for both circuit blocks 202 and 204, which may simplify configuration and integration. In some aspects, when the circuit block 202 receives a single input (such as input data 101 in FIG. 1) via input port 201A, the two input ports 201A and 201B of the circuit block 202 may be coupled to each other in one implementation. In an alternative implementation, the circuit block 202 may be simplified by removing input port 201B and removing input mux 203A, such that 201A is provided directly to 214A and 207A.

[0044]

[0052] In general, input ports 201A and 201B may receive various types of input data for processing, including signed multi-bit integer data. In one example, the input data is 8-bit two's complement input data.

[0045]

[0053] Input selectors mux 203A and 203B are configured to control which input data port is used for circuit blocks 202 and 204, respectively. For example, input selector mux 203B may select between input data port 201A (e.g., when circuit block 202 is bypassed) or 212B (e.g., when circuit blocks 202 and 204 are being processed in succession).

[0046]

[0054] Bypass selectors mux 211A and 211B are configured to control bypass function blocks 214A and 214B of circuit blocks 202 and 204, respectively. For example, when circuit block 202 is to be bypassed, bypass selector mux 211A selects bypass line 205A to provide an output to output port 212A. Similarly, when circuit block 204 is to be bypassed, bypass selector mux 211B selects bypass line 205B to provide an output to output port 216. Thus, processing with circuit blocks 202 and / or 204, as controlled by configurable bypasses 205A and 205B, produces an output at output port 216.

[0047]

[0055] 3, approximator blocks 206A and 206B may be configured with configuration parameters (e.g., function-specific coefficients as in Table 1 above) stored in registers 219A and 219B, respectively. Similarly, where approximator blocks 206A or 206B are configured to implement a lookup table-based function, table values ​​may be stored in registers 219A and 219B, respectively, as in Table 1 above.

[0048]

[0056] Each circuit block (202 and 204) further includes a minimum and maximum function block (208A for circuit block 202 and 208B for circuit block 204) for providing minimum and maximum functions. Generally, a minimum (or "min") function returns the minimum value of the provided inputs. Generally, a maximum (or "max") function returns the maximum value of the provided inputs. In one example, the minimum and maximum function blocks 208A and 208B may comprise multi-bit digital comparators that operate in either single-cycle or multi-cycle mode.

[0049]

[0057] The configuration of function blocks 214A and 214B may include a setting for function selector mux 209A and 209B, respectively. In other words, whether function blocks 214A and 214B output the min / max output from min / max blocks 208A and 208B or the value from approximator 206A and 206B is based on the configuration of function selector mux 209A and 209B. Note that in other examples, function blocks 214A and 214B may include additional function blocks that may be selected by a mux.

[0050]

[0058] As shown in Figure 1, where the approximator blocks may be processed in series, in Figure 2 the output 212A of a circuit block 202A containing a first approximator block 206A is provided as an input 212B to a circuit block 204 containing a second approximator block 206B. As in Figure 1, where bypasses 105 and 107 control the use of the first approximator block 102 and the second approximator block 104, here selectable bypasses 205A and 205B control the use of approximator blocks 206A and 206B.

[0051]

[0059] Asymmetric signal line 210 controls the configuration of circuit blocks 202 and 204. In one example, circuit blocks 202 and 204 are configured based on the value on asymmetric signal line 210 and the output values ​​from code blocks 207A and 207B based on input data received via input data port 201A. For example, the binary value received via asymmetric signal line 210 and the binary value output from code block 207A interact at AND gate 213 to control the selection of an output by mux 211A. As another example, the binary value received via asymmetric signal line 210 and the binary value output from code block 207B interact at AND gate 217 to control the selection of an input data port (such as between 201A and 212B) via mux 203B. As a further example, the binary value received via asymmetric signal line 210 and the inverted binary value output from code block 207B interact in AND gate 215 to control the selection of output mux 211B.

[0052]

[0060] Table 2 below provides a configuration summary for circuit blocks 202 and 204.

[0053] [Table 2]

[0054] Exemplary approximators for configurable nonlinear activation function circuits

[0061] FIG. 3 shows an exemplary approximator 300, which may be an example of the first approximator 102 and the second approximator 104 of FIG. 1 and / or one or both of the approximators 206A and 206B of FIG.

[0055]

[0062] An approximator 300 receives input data 302 (e.g., pre-activation data) for processing. In some examples, the input data 302 may be received from a buffer or other memory. In other examples, the input data may be received directly from the output of another processing block, such as the output of a CIM array or another vector and matrix multiply-accumulate block. Additionally, the input data may be received from another approximator, such as when the approximator 300 is the second approximator 104 in FIG. 1 or the second approximator 206B in FIG. 2.

[0056]

[0063] In some implementations, an approximator (such as 300) may include alternative processing paths. In such cases, the path logic 304 may be configured to route the input data 302 to an appropriate processing path based on, for example, configuration parameters for the approximator 300.

[0057]

[0064] In this example, processing path 306 A provides a third order approximation path for input data 302 .

[0058]

[0065] In processing path 306A, input data 302 is provided to a cubic calculator 308, which performs a cubic operation (e.g., x 3 , where x is the input data), and then the output is multiplied by a cubic parameter 312 in multiplier 310. The output of multiplier 310 is then provided to accumulator 324.

[0059]

[0066] The input data 302 is also provided to a secondary calculator 308, which performs secondary operations (e.g., x 2 , where x is the input data), the output of which is then multiplied by a quadratic parameter 318 in multiplier 316. The output of multiplier 316 is then provided to accumulator 324.

[0060]

[0067] The input data 302 is also provided to a multiplier 320, where the input data 302 is multiplied by a linear parameter 322. The output of the multiplier 320 is then provided to an accumulator 324.

[0061]

[0068] An accumulator (adder) 324 accumulates the outputs of the multipliers 310 , 316 , and 320 , as well as the intercept parameter 326 to generate output data 332 .

[0062]

[0069] The cubic parameters 312, the quadratic parameters 318, the linear parameters 322 and the intercept parameters 326 may all be stored in a memory or the like (e.g., a register) accessible to the accumulator 300. In some cases, a control unit, such as a memory control unit or a finite state machine, may configure the approximator 300 with the parameters stored in the memory. In various examples, the cubic parameters 312, the quadratic parameters 318, the linear parameters 322 and the intercept parameters 326 may be set according to the values ​​described above with respect to Table 2.

[0063]

[0070] As noted above, the order of the approximation may be configured by configuring the parameter values ​​described above. For example, for the approximator 300 to perform a quadratic approximation, the cubic parameter 312 may be set to 0. Similarly, for the approximator 300 to perform a linear approximation, the cubic parameter 312 and the quadratic parameter 318 may be set to 0.

[0064]

[0071] Some non-linear activation functions require alternative functions, such as minimum and maximum functions. Thus, processing path 306B provides a minimum and / or maximum calculator that can be used with the ReLU and ReLU6 functions described above in Table 2. Processing path 306B can be selected by path logic 304 based on configuration data for approximator 300.

[0065]

[0072] Furthermore, some nonlinear activation functions may be implemented using lookup tables, which provide a more power and time efficient mechanism for generating values ​​for some nonlinear activation functions. Thus, processing path 306C provides a lookup table-based processing path that may be used whenever, for example, a sigmoid, tanh, or similar function is used by the nonlinear activation function. Sigmoid and tanh may be calculated from each other, and thus, in some cases, only a single lookup table (e.g., sigmoid or tanh, but not both) is stored and used to implement both functions. One or more lookup tables are stored in memory and accessible to the approximator 300, which may include a memory tightly coupled to the approximator 300.

[0066] Exemplary Machine Learning Model Process Flow with Configurable Nonlinear Activation Function Circuit

[0073] FIG. 4 illustrates an example machine learning model data flow 400 implementing a configurable non-linear activation function circuit as described above with respect to FIGS. 1-3.

[0067]

[0074] In the flow 400, input data is stored in an input data buffer 401 (e.g., machine learning model layer input data) and then provided to a multiply and accumulate (MAC) circuit 402. The MAC circuit 402 may be generally configured to perform vector, array, and matrix multiply-accumulate operations, such as those frequently used in convolutional neural networks. In some examples, the MAC circuit 402 may include one or more compute-in-memory (CIM) arrays. Alternatively or additionally, the MAC circuit 402 may include a digital multiply-accumulate (DMAC). In yet further examples, the multiply-accumulate circuit 402 may be part of a machine learning accelerator, such as a neural processing unit (NPU) or another type of processing unit optimized for performing machine learning processing. In another implementation, the MAC circuit 402 may be replaced by a vector / matrix or matrix / matrix processing engine.

[0068]

[0075] The MAC circuit 402 processes input data with weight data (e.g., neural network weight data) to generate pre-activation data. For example, the MAC circuit 402 may process input data to a layer of a neural network model and generate pre-activation data as output.

[0069]

[0076] The pre-activation data is provided to a configurable non-linear activation (CNLA) function circuit 404, which is configured to generate (e.g., activate) output data based on a configured non-linear activation function. The output data may then be stored in an output data buffer 405 for later use, such as for processing another layer in a machine learning model or as output from a machine learning model.

[0070]

[0077] The CNLA function circuit 404 may be configured with configuration parameters as described with respect to the CNLA function circuit 300 in Figure 3, as well as those described in Tables 1 and 2. Additionally, the CNLA function circuit 404 may be configured to access a lookup table depending on the configured activation function.

[0071]

[0078] In some cases, the configuration parameters may include an identity of a nonlinear activation function to be applied to the input data. Based on the determined nonlinear activation function, appropriate parameters (such as those in Table 2) may be retrieved from a memory (e.g., a register) and applied to the CNLA function circuit 404, thereby configuring the CNLA function circuit 404 to process the input data. In some examples, a finite state machine, a memory control unit, or another controller may perform the configuration of the CNLA function circuit 404.

[0072]

[0079] In particular, the CNLA circuit 404 may be configured to process multiple batches of input data using the same configuration or may update the configuration of the CNLA circuit 404 for every new batch of input data. Thus, the CNLA circuit 404 provides an extremely flexible and efficient means for implementing configurable nonlinear activation for machine learning tasks, such as training and inference.

[0073] Exemplary Method for Performing Processing Using a Configurable Nonlinear Activation Function Circuit - Patent application

[0080] FIG. 5 illustrates an exemplary method 500 for performing processing using a configurable nonlinear activation function circuit.

[0074]

[0081] Method 500 begins at step 502 with selecting a non-linear activation function for application to the input data. For example, the non-linear activation function may be one of the functions set forth in Table 2, or another non-linear activation function.

[0075]

[0082] The method 500 then proceeds to step 504, which involves determining a set of parameters for a configurable nonlinear activation function circuit based on the selected nonlinear activation function. For example, the parameters for the determined nonlinear activation function may be as in Tables 1 and 2 above.

[0076]

[0083] The method 500 then proceeds to step 506, which involves processing the input data with a configurable nonlinear activation function circuit based on a set of parameters to generate output data. For example, the output data may be activation data for a layer of a neural network model.

[0077]

[0084] In some examples, the set of parameters includes a combination of one or more gain parameters, a constant parameter, and one or more approximation functions for application to the input data via the configurable nonlinear activation function circuit. For example, the set of parameters may be as described above with respect to Figures 1 and 2 and in Table 1.

[0078]

[0085] In some examples, the method 500 further includes retrieving the set of parameters from a memory based on the determined nonlinear activation function. In some examples, the memory may be one or more registers that store the parameter values.

[0079]

[0086] In some examples, the configurable nonlinear activation function circuit includes a first approximator configured to approximate a first function of the one or more approximation functions, a second approximator configured to approximate a second function of the one or more approximation functions, a first gain multiplier configured to multiply a first gain value based on the one or more gain parameters, and a constant adder configured to add a constant value, such as shown and described with respect to FIG.

[0080]

[0087] In some examples, the configurable nonlinear activation function circuit includes a first bypass configured to bypass the first approximator. In some examples, the configurable nonlinear activation function circuit includes a second bypass configured to bypass the second approximator. In some examples, the configurable nonlinear activation function circuit includes an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0081]

[0088] In some examples, at least one of the first approximator and the second approximator is a third order approximator. In some examples, the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator. In some examples, the other of the first approximator and the second approximator is configured to implement a min or max function, such as shown in FIG. 3 for path 306B. In some examples, the other of the first approximator and the second approximator is configured to access a lookup table for approximation values, such as shown in FIG. 3 for path 306C.

[0082]

[0089] In some examples, both the first and second approximators are third order approximators.

[0083]

[0090] It should be noted that FIG. 5 is just one example, and that in other examples, methods such as those described herein may be implemented using more, fewer, and / or different steps.

[0084] Exemplary Pipelined Approximation for Nonlinear Functions

[0091] 6 shows an example of a pipelined continuous linear approximation architecture 600 that can be used to implement nonlinear functions used in machine learning model acceleration. For example, architecture 600 can be used to implement approximators 102 and 104 described with respect to FIG. 1 and 206A and 206B described with respect to FIG.

[0085]

[0092] In general, Figure 6 shows an input x flowing through linear approximators 604A-C in a pipelined fashion. The architecture 600 is a power and space efficient way to implement a cubic approximation using successive linear approximation blocks 604A-C. In particular, the input x is processed in the linear approximator block 604A to determine the output y1 = Ax + B, where A and B are coefficients 605A used by the linear approximator block 604A.

[0086]

[0093] Next, y1 is provided as an input to a linear approximator block 604B to determine the output y2=x(y1)+C, where C is a coefficient 605B used by the linear approximator block 604B. Note that x(y1) denotes y1 multiplied by x.

[0087]

[0094] Next, y2 is provided as an input to a linear approximator block 604C to determine the output y3=x(y2)+D, where D is a coefficient 605C used by the linear approximator block 604C. Expanding y3 based on successive linear approximations gives:

[0088]

number

[0089]

[0095] This is a third-order approximation based on the input x. In particular, the continuous linear approximation technique implemented in architecture 600 reduces the number of multipliers required, which reduces the complexity and power used compared to conventional third-order approximator architectures. Furthermore, architecture 600 enables a pipelined implementation, which produces a new output every cycle in the steady state, thus improving throughput.

[0090]

[0096] To further improve processing efficiency, the coefficients A, B, C, and D (605A-C) may be stored in registers, for example, rather than SRAM or DRAM, which improves the power and latency performance of the architecture 600. For example, read power usage from the host system SRAM may be avoided by using local registers to store the coefficient values. The register approach also supports a pipelined architecture by avoiding additional cycles to read and transport the coefficient values ​​from a remote host system memory (e.g., DRAM). In various aspects, the pipeline registers may be internal or external to the linear approximators 604A-C.

[0091]

[0097] In some implementations, as shown with respect to architecture 600, the coefficients may be determined based on an approximation of a non-uniformly segmented non-linear function, as described in one example with respect to Figure 7. In such an embodiment, region finder 602 may determine appropriate coefficients based on a region (or segment) of the non-uniformly segmented approximation according to input x. Thus, Figure 6 shows the output of the region finder, "REGION_SEL," being passed through various stages of the linear approximation in a pipelined manner.

[0092]

[0098] Note that the "region" being found by the region finder 602 is a segment of the "domain" of the original nonlinear function. That is, the nonlinear function may be defined over a domain from negative infinity to positive infinity, and that domain may be partitioned into segments. In various aspects described herein, the segments are of non-uniform length. For each such segment, coefficients are chosen to best approximate the function within that segment. FIG. 8 shows an example circuit implementation with a region finder element 809, where "cutoff" is used to describe the segment end points.

[0093]

[0099] In various aspects, the region finder 602 may be configured to select the entire set of coefficients to be used for the entire approximation. In the illustrated example, since the approximation is performed serially in a pipelined manner, the determined region will be registered live with each approximator data path, and thus the determined region may be used for subsequent coefficient lookups at each stage (e.g., for the linear approximators 604A-C).

[0094]

[0100] The region finder 602 generally allows various offline algorithms to be used to search and set the region selection, so the implementation need not be limited by how the region is found. For example, a cubic spline algorithm may be used to ensure continuous differentiation across the region boundaries, which is beneficial for model training. In another example, the mean squared error may be minimized pointwise (based on a subsequent search for the best cubic coefficient given the region under consideration). This second approach may achieve better accuracy of the calculations (achieving better accuracy for inference) at the expense of continuous differentiation. Yet another approach is to set the region based on when the second or third derivative is zero, but this approach may not use the full set of possible regions, as more regions will produce less error.

[0095]

[0101] Finally, the architecture 600 is configured to exploit the odd and even symmetry properties of common neural network nonlinear functions, such as sigmoid and hyperbolic tangent (tanh), to beneficially reduce circuit area and power during processing. Beneficially, exploiting such symmetry allows for reducing the number of coefficients that need to be stored to approximate a nonlinear function, for example, because a simple sign inversion can emulate both sides (e.g., positive and negative) of the function from the origin without the need to store coefficients for both sides. In some implementations, the approximation result is calculated only for positive inputs, and then the same result is used for negative inputs with the same magnitude (even symmetry), or the sign of the result can be inverted in the case of functions with odd symmetry.

[0096]

[0102] In the illustrated example, a sign and offset corrector component 606 is configured to take as inputs a third order approximation y3, an original input x (e.g., to determine its sign), and an offset value 607, and apply sign and / or offset correction to generate a final output 608. For example, the final output 608 may be calculated as offset(607)+ / -y3. As with coefficients 605A-C, offset 607 may be stored in a local register for efficiency and speed.

[0097]

[0103] In general, the offset 607 may be used for functions that are symmetric except for those centered around x=0. One example is the sigmoid, which is symmetric about the line y=0.5. In such a case, a "shifted" sigmoid is first fitted, which is odd-symmetric about the x-axis, and then an offset of 0.5 is added to the result.

[0098]

[0104] Note that FIG. 6 illustrates an example architecture 600 including three linear approximators (604A-C) that may be used to approximate a third-order nonlinear function. In other aspects, two linear approximators may be used when a second-order nonlinear function is intended to be approximated. Although not shown in FIG. 6, in some aspects, the control logic may switch the output of a linear approximator, such as linear approximator 604B, to bypass a subsequent linear approximator (such as linear approximator 604C) when only a second-order function needs to be approximated. In such a case, the bypass may take the output y3 of linear approximator 604B and route it to sign and offset corrector component 606.

[0099]

[0105] FIG. 7 shows an example 700 of a hyperbolic tangent (tanh) nonlinear activation function with non-uniform segments for defining the approximated function output.

[0100]

[0106] In the illustrated example, a first segment 702A is defined from the origin (x=0) to a first point along the curve 701. As shown, a segment such as 702B will have a longer length along the curve 701 if the slope is more constant across the input values ​​in that segment, and a segment such as 702C will have a shorter length along the curve 701 if the curve is changing slope more rapidly across the input values ​​in that segment. Using non-uniform segments beneficially allows for defining fewer segments spanning the output value range for a non-linear function while still maintaining a similar output resolution compared to the actual function. Traditionally, to maintain resolution, curves such as 701 were uniformly segmented, which meant that many segments had exactly similar output values, thus creating redundant values ​​that were stored in memory.

[0101]

[0107] Note that each segment may be generally defined by a start point and an end point along the curve 701. For example, a starting input value x start and the end input value x end may define segments. Each of these segments may bestart ≦× <x end etc., may be associated with a set of coefficients for use in approximating a hyperbolic tangent function-based input value that falls within a defined range. To further conserve memory, the first segment 702A may be assumed to have a starting point (origin) at x=0, so that only the end point of the segment needs to be stored. Similarly, the last segment is defined only by the end point of the penultimate segment. That is, all inputs greater than or equal to the end point of the penultimate segment are considered part of the last segment.

[0102]

[0108] For example, the following table gives examples of non-uniform segments that may be stored in registers for a tanh function to be approximated: When the input value x falls within a particular segment (or region), the coefficients for that segment may be retrieved and used to generate the approximated function value.

[0103] [Table 3]

[0104]

[0109] Note that the hyperbolic tangent function curve 701 is an example of a function with odd symmetry. That is, the tanh output value for a positive input (e.g., 0.2 for point 704A) is equal to the sign-negated output for a negative input of the same magnitude (e.g., −0.2 for 704B). Thus, non-uniform segments need only be defined for the positive input value range for the hyperbolic tangent function curve 701, and a sign corrector component, such as 606 in FIG. 6, can appropriately invert the sign of the final output (e.g., 608 in FIG. 6) based on the sign of the input value (x).

[0105]

[0110] FIG. 8 shows an example of a circuit 800 for implementing a pipelined continuous linear approximation architecture as described above with respect to FIG. 6. In particular, FIG. 8 shows the pipelining features of the design, including how the output of region finder 809 and the sign bits are passed through the pipeline for efficient processing.

[0106]

[0111] The illustrated example includes four stages 806A - 806D, three of which include linear approximators (804A - 804C respectively) and one is the sign and offset correction stage 806D.

[0107]

[0112] In the illustrated example, various operation blocks are shown. "SAT" represents the "saturation" block. Saturation is useful when reducing the number of integer bits used to represent a value. When a saturation module is used to reduce an input having M integer bits to an output having N integer bits (where N < M), input values greater than what can be represented by N integer bits are "saturated" to either the maximum positive N - bit number (for positive inputs) or the minimum negative N - bit number (for negative inputs). For example, if [x > 0111...11], output y(0111...11), or if [x ≤ 1000...00], output y(1000...00).

[0108]

[0113] "ABS" represents a block that calculates the absolute value (or magnitude) of the input.

[0109]

[0114] "COMP" represents a comparator block that acts as a region finder using a programmable cutoff. The cutoff defines the end points of various segments (and, indirectly, the start point for the last segment). The output of the COMP block 809 is an identifier indicating which region (segment) the current input belongs to. This identifier is used to select the appropriate set of coefficients for approximating the non - linear function within that segment.

[0110]

[0115] In an exemplary implementation, eleven cutoff values are used to define a total of twelve segments. By leveraging the symmetry property of the non-linearity function, the COMP module operates only on the absolute value of the input. The following algorithm describes the operation of the COMP module. IF 0 <= ABS(x) < CUTOFF1: Select the segment 0 coefficient ELSE IF (CUTOFF1 <= ABS(x) < CUTOFF2): Select the segment 1 coefficient ELSE IF (CUTOFF2 <= ABS(x) < CUTOFF3): Select the segment 2 coefficient ... ELSE IF (CUTOFF10 <= ABS(x) < CUTOFF11): Select the segment 10 coefficient ELSE IF (CUTOFF11 <= ABS(x)): Select the segment 11 coefficient

[0116] "TRUNC" represents a truncation block, which is configured to reduce the precision of a fraction represented in two's complement form by omitting the least significant bits. For example, to reduce the input from having M fractional bits to having N fractional bits, the least significant (M - N) bits are omitted (truncated).

[0111]

[0117] "SYMMSAT" represents a "symmetric saturation" block. A conventional saturation (SAT) module with N integer bits saturates to either the largest positive value (2 N-1 - 1) or the smallest negative value (-2 N-1 ). In the symmetric saturation module, a negative input saturates instead to a value that is 1 greater than the smallest negative value (e.g., (-2 N-1 + 1)). In this way, the number of distinct negative and positive outputs is equal. For example, if [x > 0111...11], output y(0111...11), or if [x <= 1000...00], output y(1000...01).

[0112]

[0118] "SE" stands for sign extension block, which is useful when increasing the number of integer bits used to represent a value. For example, when a value represented with N integer bits is required to be represented with M integer bits (M>N), the sign bit is replicated (MN) times.

[0113]

[0119] "TC" represents a block used to convert a positive number to a negative number by taking the two's complement of the number.

[0114]

[0120] Finally, "ZF" stands for "zero-fill" block, which is used when a number with N fractional bits is desired to be represented as a number with M fractional bits, where M>N, by appending (MN) zeros to the least significant positions.

[0115]

[0121] Figure 9 shows an example of a linear approximator circuit 900, which may be an example of the linear approximator blocks 804A-C in circuit 800 of Figure 8. As shown, the linear approximator circuit includes a multiplier element 902 that takes as inputs a stage input and a coefficient 904, and an adder element 906 that takes as inputs the output of the multiplier element after being processed by a truncation block and a saturation block, and a second coefficient 908. Exemplary Processing System

[0122] FIG. 10 illustrates an example processing system 1000 that may be configured to perform the methods described herein, such as with respect to FIGS.

[0116]

[0123] The processing system 1000 includes a central processing unit (CPU) 1002, which may be a multi-core CPU in some examples. Instructions executed in the CPU 1002 may be loaded, for example, from a program memory associated with the CPU 1002 or may be loaded from a memory partition 1024.

[0117]

[0124] The processing system 1000 also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, a neural processing unit (NPU) 1008, a multimedia processing unit 1010, and wireless connectivity components 1012.

[0118]

[0125] An NPU, such as 1008, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic to run machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. An NPU may alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0119]

[0126] NPUs, such as 1008, may be configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, they may be part of a dedicated machine learning accelerator device.

[0120]

[0127] NPUs can be optimized for training or inference, or in some cases configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still be performed largely independently.

[0121]

[0128] NPUs designed to accelerate training are configured to accelerate the optimization of new models, which is a highly compute-intensive operation that generally involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, to improve model performance. Generally, optimizing based on erroneous predictions involves backpropagating through layers of the model and determining gradients to reduce prediction errors.

[0122]

[0129] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input new data and rapidly process it through already-trained models to generate model outputs (e.g., inferences).

[0123]

[0130] In some embodiments, the NPU 1008 may be implemented as part of one or more of the CPU 1002, GPU 1004, and / or DSP 1006.

[0124]

[0131] In some embodiments, the wireless connectivity component 1012 may include subcomponents for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 1012 is further connected to one or more antennas 1014.

[0125]

[0132] The processing system 1000 may also include one or more sensor processing units 1016 associated with any type of sensor, one or more image signal processors (ISPs) 1018 associated with any type of image sensor, and / or a navigation processor 1020, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0126]

[0133] The processing system 1000 may also include one or more input and / or output devices 1022, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, or the like.

[0127]

[0134] In some examples, one or more of the processors of processing system 1000 may be based on the ARM or RISC-V instruction set.

[0128]

[0135] The processing system 1000 also includes various circuits according to various embodiments described herein.

[0129]

[0136] In this example, the processing system 1000 includes a compute-in-memory (CIM) circuit 1026, which may be configured to implement an efficient multiply-accumulate (MAC) function for processing machine learning model data. The processing system 1000 further includes a configurable nonlinear activation (CNLA) function circuit 1028. In some cases, the CNLA function circuit 1028 may be like the CNLA function circuit 200 described with respect to Figures 2, 3, 6, 8, and 9. The CNLA function circuit 1028, as well as others not shown, may be configured to implement various aspects of the methods described herein, such as the methods 400 and 500 with respect to Figures 4 and 5, respectively.

[0130]

[0137] In some examples, the CNLA function circuit 1028 may be implemented as part of another processing unit, such as the CPU 1002, the GPU 1004, the DSP 1006, or the NPU 1008.

[0131]

[0138] Processing system 1000 also includes memory 1024, which represents one or more static and / or dynamic memories, such as dynamic random access memories, flash-based static memories, etc. In this example, memory 1024 includes computer-executable components that may be executed by one or more of the above-mentioned components of processing system 1000.

[0132]

[0139] In particular, in this example, memory 1024 includes a determination component 1024A, a configuration component 1024B, a processing component 1024C, a retrieval component 1024D, nonlinear activation function parameters 1024E, lookup table(s) 1024F, and model parameters 1024G (e.g., weights, biases, and other machine learning model parameters). One or more of the illustrated components, as well as other components not illustrated, may be configured to implement various aspects of the methods described herein.

[0133]

[0140] Generally, the processing system 1000 and / or its components may be configured to perform the methods described herein.

[0134]

[0141] Notably, in other embodiments, aspects of the processing system 1000 may be omitted, such as when the processing system 1000 is a server computer, etc. For example, the multimedia components 1010, the wireless connectivity 1012, the sensors 1016, the ISP 1018, and / or the navigation components 1020 may be omitted in other embodiments. Additionally, aspects of the processing system 1000 may be distributed.

[0135]

[0142] It should be noted that FIG. 10 is just one example, and that in other examples, alternative processing systems having more, fewer and / or different components may be used.

[0136] Example clauses

[0143] Example implementations are described in the following numbered clauses.

[0137]

[0144] Clause 1: A processor comprising a configurable nonlinear activation function circuit, the configurable nonlinear activation function circuit configured to determine a nonlinear activation function for application to input data, determine a set of parameters for the nonlinear activation function based on the determined nonlinear activation function, and generate output data based on application of the set of parameters for the nonlinear activation function.

[0138]

[0145] Clause 2: The processor of clause 1, wherein the configurable nonlinear activation function circuit comprises a first approximator configured to approximate a first function using one or more first function parameters of the set of parameters, a second approximator configured to approximate a second function using one or more second function parameters of the set of parameters, a gain multiplier configured to multiply a gain value based on one or more gain parameters of the set of parameters, and a constant adder configured to add a constant value based on a constant parameter of the set of parameters.

[0139]

[0146] Clause 3: The processor of clause 2, wherein at least one of the first approximator and the second approximator is a third order approximator.

[0140]

[0147] Clause 4: The processor of clause 3, wherein the other one of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator.

[0141]

[0148] Clause 5: The processor of clause 2, wherein both the first approximator and the second approximator are third order approximators.

[0142]

[0149] Clause 6: The processor of clause 3, wherein the other one of the first approximator and the second approximator is configured to access a lookup table for the approximation value.

[0143]

[0150] Clause 7: The processor of clause 3, wherein the other one of the first approximator and the second approximator is configured to implement a minimum or maximum function.

[0144]

[0151] Clause 8: The processor of clause 2, wherein the determined nonlinear activation function comprises a swish function, the gain parameter comprises a dependent parameter value of 1 and an independent parameter value of 0, the constant value is 0, the first function is quadratic, and the second function is a sigmoid look-up table.

[0145]

[0152] Clause 9: The processor of clause 2, wherein the determined nonlinear activation function comprises a hard swish function, the gain parameter comprises a dependent parameter value of 1 / 6 and an independent parameter value of 0, the constant value is 3, the first function is a max function, and the second function is a min function.

[0146]

[0153] Clause 10: The processor of clause 2, wherein the determined nonlinear activation function comprises a hyperbolic tangent (tanh) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is quadratic, and the second function is a tanh lookup table.

[0147]

[0154] Clause 11: The processor of clause 2, wherein the determined nonlinear activation function comprises a sigmoid function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is linear, and the second function is a sigmoid lookup table.

[0148]

[0155] Clause 12: The processor of clause 2, wherein the determined nonlinear activation function comprises a Gaussian Error Linear Unit (GELU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 1, the first function is cubic, and the second function is a tanh lookup table.

[0149]

[0156] Clause 13: The processor of clause 2, wherein the determined nonlinear activation function comprises a rectified linear unit (ReLU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is quadratic, and the second function is a max function.

[0150]

[0157] Clause 14: The processor of clause 2, wherein the determined nonlinear activation function comprises a rectified linear unit 6 (ReLU6) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a max function, and the second function is a min function.

[0151]

[0158] Clause 15: The processor of clause 2, wherein the determined nonlinear activation function comprises an exponential linear unit (ELU) function, the gain parameter has a dependent parameter value of 0 and an independent parameter value of α, the constant value is 0, the first function is quadratic if the input data value is ≧0 or is bypassed if the input data value is <0, and the second function is bypassed if the input data value is ≧0 or is an exponential lookup table if the input data value is <0.

[0152]

[0159] Clause 16: A processor described in any one of clauses 1 to 15, further comprising an input memory buffer configured to store one or more outputs received from the processing circuit as input data, and an output memory buffer configured to store generated output data for output from the configurable nonlinear activation function circuit.

[0153]

[0160] Clause 17: The processor of any one of clauses 1 to 16, further comprising a compute-in-memory array configured to provide input data to the configurable non-linear activation function circuit.

[0154]

[0161] Clause 18: A method for processing input data with a configurable non-linear activation function circuit, the method comprising: determining a non-linear activation function for application to the input data; determining a set of parameters for the configurable non-linear activation function circuit based on the determined non-linear activation function; and processing the input data using the configurable non-linear activation function circuit based on the set of parameters to generate output data.

[0155]

[0162] Clause 19: The method of clause 18, further comprising retrieving a set of parameters from a memory based on the determined nonlinear activation function.

[0156]

[0163] Clause 20: The method of clause 18, wherein the set of parameters includes a combination of one or more gain parameters, a constant parameter, and one or more approximation functions for application to the input data via the configurable nonlinear activation function circuit.

[0157]

[0164] Clause 21: The method of clause 20, wherein the configurable nonlinear activation function circuit comprises a first approximator configured to approximate a first function of the one or more approximation functions, a second approximator configured to approximate a second function of the one or more approximation functions, a first gain multiplier configured to multiply a first gain value based on one or more gain parameters, and a constant adder configured to add a constant value based on a constant parameter.

[0158]

[0165] Clause 22: The method of clause 21, wherein the configurable nonlinear activation function circuit further comprises a first bypass configured to bypass the first approximator, a second bypass configured to bypass the second approximator, and an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0159]

[0166] Clause 23: The method of clause 22, wherein at least one of the first approximator and the second approximator is a third order approximator.

[0160]

[0167] Clause 24: The method of clause 23, wherein the other one of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator.

[0161]

[0168] Clause 25: The method of clause 23, wherein both the first approximator and the second approximator are third order approximators.

[0162]

[0169] Clause 26: The method of clause 23, wherein the other one of the first approximator and the second approximator is configured to access a lookup table for the approximation value.

[0163]

[0170] Clause 27: The method of clause 23, wherein the other one of the first approximator and the second approximator is configured to implement a min or max function.

[0164]

[0171] Clause 28: The method of clause 21, wherein the determined nonlinear activation function comprises a swish function, the gain parameter comprises a dependent parameter value of 1 and an independent parameter value of 0, the constant value is 0, the first function is quadratic and the second function is a sigmoidal lookup table.

[0165]

[0172] Clause 29: The method of clause 21, wherein the determined nonlinear activation function comprises a hard swish function, the gain parameter comprises a dependent parameter value of 1 / 6 and an independent parameter value of 0, the constant value is 3, the first function is a max function and the second function is a min function.

[0166]

[0173] Clause 30: The method of clause 21, wherein the determined nonlinear activation function comprises a Gaussian Error Linear Unit (GELU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 1, the first function is cubic, and the second function is a tanh lookup table.

[0167]

[0174] Clause 31: The method of clause 21, wherein the determined nonlinear activation function comprises a hyperbolic tangent (tanh) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is quadratic, and the second function is a tanh lookup table.

[0168]

[0175] Clause 32: The method of clause 21, wherein the determined nonlinear activation function comprises a sigmoid function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is linear, and the second function is a sigmoid lookup table.

[0169]

[0176] Clause 33: The method of clause 21, wherein the determined nonlinear activation function comprises a rectified linear unit (ReLU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is quadratic and the second function is a max function.

[0170]

[0177] Clause 34: The method of clause 21, wherein the determined nonlinear activation function comprises a rectified linear unit 6 (ReLU6) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a max function and the second function is a min function.

[0171]

[0178] Clause 35: The method of clause 21, wherein the determined nonlinear activation function comprises an exponential linear unit (ELU) function, the gain parameter has a dependent parameter value of 0 and an independent parameter value of α, the constant value is 0, the first function is quadratic if the input data value is ≧0 or is bypassed if the input data value is <0, and the second function is bypassed if the input data value is ≧0 or is an exponential lookup table if the input data value is <0.

[0172]

[0179] Clause 36: The method of clause 21, further comprising receiving input data from a compute-in-memory (CIM) array.

[0173]

[0180] Clause 37: A configurable nonlinear activation function circuit configured to process a nonlinear activation function according to any of the configurations in Table 1.

[0174]

[0181] Clause 38: A circuit block comprising a configurable nonlinear activation function circuit and a selectable bypass.

[0175]

[0182] Clause 39: The circuit block of clause 38, wherein the configurable nonlinear activation function circuit is configured to process the nonlinear activation function according to any of the configurations in Table 1.

[0176]

[0183] Clause 40: The circuit block of clause 38, wherein the circuit block may be configured according to any of the configurations in Table 2.

[0177]

[0184] Clause 41: A processor comprising a configurable nonlinear activation function circuit, the configurable nonlinear activation function circuit configured to determine a set of parameters for a selected nonlinear activation function based on a selected nonlinear activation function, and to generate output data based on application of the set of parameters for the selected nonlinear activation function, wherein the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two continuous linear approximators, each linear approximator of the at least two continuous linear approximators configured to approximate a linear function using one or more function parameters of the set of parameters.

[0178]

[0185] Clause 42: The processor of clause 41, wherein each linear approximator of the at least two continuous linear approximators comprises a stage input, a coefficient input, a constant input, and a stage output.

[0179]

[0186] Clause 43: A processor according to any one of clauses 41 to 42, wherein at least one non-linear approximator comprises a third order approximator comprising three successive linear approximators.

[0180]

[0187] Clause 44: A processor according to any one of clauses 41 to 43, wherein at least one non-linear approximator comprises a quadratic approximator comprising two successive linear approximators.

[0181]

[0188] Clause 45: The processor of clause 42, wherein at least one nonlinear approximator further comprises a region finder component configured to determine an input value region, and each linear approximator of the at least two continuous linear approximators is further configured to determine coefficient inputs and constant inputs based on the input value region.

[0182]

[0189] Clause 46: A processor described in any one of clauses 41 to 45, wherein at least one nonlinear approximator further comprises a sign and offset corrector component configured to correct a stage output from at least one linear approximator of the at least two successive linear approximators.

[0183]

[0190] Clause 47: The processor of clause 46, wherein the sign and offset corrector component is further configured to invert a sign of a stage output to correct the stage output from at least one linear approximator of the at least two continuous linear approximators.

[0184]

[0191] Clause 48: The processor of clause 46, wherein the sign and offset corrector component is further configured to add an offset to a stage output from at least one linear approximator of the at least two continuous linear approximators to correct the stage output.

[0185]

[0192] Clause 49: A processor as described in any one of clauses 41 to 48, wherein each linear approximator of the at least two continuous linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of a function approximation, so that each linear approximator of the at least two continuous linear approximators approximates a linear function using one or more function parameters of the set of parameters.

[0186]

[0193] Clause 50: A method for processing data using a configurable nonlinear activation function circuit, comprising: determining a set of parameters for a selected nonlinear activation function based on a selected nonlinear activation function; and generating output data based on application of the set of parameters for the selected nonlinear activation function, wherein the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two continuous linear approximators, each linear approximator of the at least two continuous linear approximators configured to approximate a linear function using one or more function parameters of the set of parameters.

[0187]

[0194] Clause 51: The method of clause 50, wherein each linear approximator of the at least two continuous linear approximators comprises a step input, a coefficient input, a constant input, and a step output.

[0188]

[0195] Clause 52: The method of any one of clauses 50 or 51, wherein at least one non-linear approximator comprises a third order approximator comprising three successive linear approximators.

[0189]

[0196] Clause 53: The method of any one of clauses 50 to 52, wherein at least one non-linear approximator comprises a quadratic approximator comprising two successive linear approximators.

[0190]

[0197] Clause 54: The method of clause 51, wherein at least one nonlinear approximator further comprises a region finder component configured to determine an input value region, and each linear approximator of the at least two continuous linear approximators is further configured to determine coefficient inputs and constant inputs based on the input value region.

[0191]

[0198] Clause 55: The method of any one of clauses 50 to 54, further comprising correcting a stage output from at least one linear approximator of the at least two successive linear approximators using a sign and offset corrector component.

[0192]

[0199] Clause 56: The method of clause 55, further comprising inverting the sign of the stage output using a sign and offset corrector component to correct the stage output from at least one linear approximator of the at least two successive linear approximators.

[0193]

[0200] Clause 57: The method of clause 55, further comprising adding an offset to the stage output using a sign and offset corrector component to correct the stage output from at least one linear approximator of the at least two successive linear approximators.

[0194]

[0201] Clause 58: The method of any one of clauses 50 to 57, wherein each linear approximator of the at least two continuous linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of the function approximation, such that each linear approximator of the at least two continuous linear approximators approximates the linear function using one or more function parameters of the set of parameters.

[0195]

[0202] Clause 59: A processing system comprising a memory having computer-executable instructions and one or more processors, the one or more processors being configured to execute the computer-executable instructions to cause the processing system to perform a method according to any one of clauses 18 to 36 or 50 to 58.

[0196]

[0203] Clause 60: A processing system comprising means for carrying out the method according to any one of clauses 18 to 36 or 50 to 58.

[0197]

[0204] Clause 61: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of clauses 18 to 36 or 50 to 58.

[0198]

[0205] Clause 62: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any one of clauses 18 to 36 or 50 to 58.

[0199] Additional Considerations

[0206] The above description is provided to enable those skilled in the art to practice the various embodiments described herein. The examples described herein are not intended to limit the scope, applicability, or embodiments described in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of the elements described without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented, or a method may be practiced, using any number of aspects described herein. Furthermore, the scope of the disclosure is intended to cover such apparatus or methods implemented using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be implemented by one or more elements of a claim.

[0200]

[0207] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0201]

[0208] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0202]

[0209] The term "determining" as used herein encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Additionally, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0203]

[0210] As used herein, the term "connected to," in the context of sharing electronic signals and data between elements described herein, may generally refer to data communication between the respective elements connected to one another. In some cases, the elements may be directly connected to one another, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between the respective elements directly connected to one another. In other cases, the elements may be indirectly connected to one another, such as via one or more data buses or similar shared circuitry and / or integrated circuit elements for communicating signals and data between the respective elements indirectly connected to one another.

[0204]

[0211] The methods disclosed herein comprise one or more steps or actions for achieving the method. The steps and / or actions of the methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. Those means may include various (one or more) hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0205]

[0212] The following claims are not limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. In the claims, reference to an element in the singular does not mean "the one and only" unless expressly stated as such, but means "one or more." Unless otherwise expressly stated, the term "several" refers to one or more. No claim element is to be construed under the provisions of 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are encompassed by the claims. Moreover, nothing disclosed herein is made public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. Configurable nonlinear activation function circuits 1. A processor comprising: determining a set of parameters for the selected nonlinear activation function based on the selected nonlinear activation function; generating output data based on application of the set of parameters for the selected nonlinear activation function; configured to: Here, the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two successive linear approximators; a processor, wherein each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the set of parameters.

2. each linear approximator of the at least two successive linear approximators: Step input and Coefficient input and Constant input and Step output and The processor of claim 1 , comprising:

3. The processor of claim 1 , wherein the at least one non-linear approximator comprises a third order approximator comprising three successive linear approximators.

4. The processor of claim 1 , wherein the at least one nonlinear approximator comprises a quadratic approximator comprising two successive linear approximators.

5. the at least one nonlinear approximator further comprising a region finder component configured to determine a region of input values; each linear approximator of the at least two continuous linear approximators is further configured to determine the coefficient input and the constant input based on the input value range; The processor of claim 2 .

6. 2. The processor of claim 1, wherein the at least one non-linear approximator further comprises a sign and offset corrector component configured to modify a stage output from at least one linear approximator of the at least two continuous linear approximators, and preferably the sign and offset corrector component is further configured to invert the sign of the stage output from the at least one linear approximator of the at least two continuous linear approximators in order to modify the stage output from the at least one linear approximator of the at least two continuous linear approximators, or to add an offset to the stage output from the at least one linear approximator of the at least two continuous linear approximators in order to modify the stage output from the at least one linear approximator of the at least two continuous linear approximators.

7. 2. The processor of claim 1, wherein each linear approximator of the at least two continuous linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of a function approximation for each linear approximator of the at least two continuous linear approximators to approximate a linear function using one or more function parameters of the set of parameters.

8. 1. A method for processing data using a configurable nonlinear activation function circuit, comprising: determining a set of parameters for the selected nonlinear activation function based on the selected nonlinear activation function; generating output data based on application of the set of parameters for the selected nonlinear activation function; Here, the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two successive linear approximators; wherein each linear approximator of said at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of said set of parameters.

9. each linear approximator of the at least two successive linear approximators: Step input and Coefficient input and Constant input and Step output and The method of claim 8 , comprising:

10. The method of claim 8 , wherein the at least one non-linear approximator comprises a third order approximator comprising three successive linear approximators.

11. The method of claim 8 , wherein the at least one non-linear approximator comprises a quadratic approximator comprising two successive linear approximators.

12. the at least one nonlinear approximator further comprising a region finder component configured to determine a region of input values; each linear approximator of the at least two continuous linear approximators is further configured to determine the coefficient input and the constant input based on the input value range; 10. The method of claim 9.

13. 9. The method of claim 8, further comprising modifying a staged output from at least one linear approximator of the at least two continuous linear approximators using a sign and offset corrector component, preferably wherein the sign and offset corrector component is further configured to invert the sign of the staged output from the at least one linear approximator of the at least two continuous linear approximators in order to modify the staged output from the at least one linear approximator of the at least two continuous linear approximators, or to add an offset to the staged output from the at least one linear approximator of the at least two continuous linear approximators in order to modify the staged output from the at least one linear approximator of the at least two continuous linear approximators.

14. 9. The method of claim 8, wherein each linear approximator of the at least two continuous linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of a function approximation for each linear approximator of the at least two continuous linear approximators to approximate a linear function using one or more function parameters of the set of parameters.

15. 15. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor comprising a configurable non-linear activation function circuit, cause the processor to perform the method of any one of claims 8 to 14.

Images