Configurable non-linear activation function circuit

Abstract

Certain aspects of the present disclosure provide a processor comprising: a configurable non-linear activation function circuit configured to: determine a set of parameters for a non-linear activation function based on a selected non-linear activation function; and generating output data based on the application of the parameter set for the non-linear activation function, in which: the configurable non-linear activation function circuit comprises at least one non-linear approximator comprising at least two successive linear approximators, and each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the parameter set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application Ser. No. 17 / 807,125, filed on June 15, 2022, which is a continuation-in-part of U.S. patent application Ser. No. 17 / 467,079, filed on September 3, 2021, the entire contents of which are incorporated herein by reference.

[0003] introduction

[0004] Aspects of the present disclosure relate to processing non-linear activation functions for machine learning models, and more particularly to configurable non-linear activation function circuits.

[0005] Machine learning is generally the process of producing a trained model (e.g., an artificial neural network) that represents a generalized fit to a training data set. Applying the trained model to new data enables inferences to be produced, which can be used to gain insights about the new data.

[0006] As the use of machine learning has proliferated to implement various machine learning (or artificial intelligence) tasks, there has been a need to more efficiently process machine learning model data. In some cases, dedicated hardware (such as a machine learning (or artificial intelligence) accelerator or processor or similar circuit) can be used to enhance the ability of a processing system to process machine learning model data. For example, processing data using a nonlinear activation function can be assigned to a processor other than the main matrix multiplication processor. However, distributing various aspects of processing machine learning models across different processing devices can result in latency, memory usage, power usage, and other processing losses.

[0007] Therefore, there is a need for improved techniques for processing machine learning model data utilizing non-linear activation functions.

[0008] Brief Overview

[0009] Certain aspects provide a processor comprising: a configurable non-linear activation function circuit configured to: determine a non-linear activation function for application to input data; determine a set of parameters for the non-linear activation function based on the determined non-linear activation function; and generate output data based on application of the set of parameters for the non-linear activation function.

[0010] A further aspect provides a method for processing input data by a configurable nonlinear activation function circuit, comprising: determining a nonlinear activation function for application to the input data; determining a parameter set for the configurable nonlinear activation function circuit based on the determined nonlinear activation function; and processing the input data using the configurable nonlinear activation function circuit based on the parameter set to generate output data.

[0011] A further aspect provides a processor comprising: a configurable nonlinear activation function circuit configured to: determine a parameter set for a nonlinear activation function based on a selected nonlinear activation function; and generate output data based on application of the parameter set for the nonlinear activation function, wherein: the configurable nonlinear activation function circuit comprises at least one nonlinear approximator, the at least one nonlinear approximator comprises at least two consecutive linear approximators, and each linear approximator of the at least two consecutive linear approximators is configured to approximate a linear function using one or more function parameters in the parameter set.

[0012] A further aspect provides a method for processing data using a configurable nonlinear activation function circuit, comprising: determining a parameter set for a nonlinear activation function based on a selected nonlinear activation function; and generating output data based on application of the parameter set for the nonlinear activation function, wherein: the configurable nonlinear activation function circuit includes at least one nonlinear approximator, the at least one nonlinear approximator includes at least two consecutive linear approximators, and each linear approximator of the at least two consecutive linear approximators is configured to approximate a linear function using one or more function parameters in the parameter set.

[0013] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising means for performing the aforementioned methods and those further described herein.

[0014] The following description and related drawings set forth in detail certain illustrative features of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings depict certain aspects of the one or more embodiments and therefore are not to be considered limiting the scope of the disclosure.

[0017] Figure 1 An example configurable nonlinear activation (CNLA) function circuit is depicted.

[0018] Figure 2 Describes the use of Figure 1 Example circuit blocks described that may bypass an approximator block.

[0019] Figure 3 An example approximator is depicted.

[0020] Figure 4 Depicts an example machine learning model process flow.

[0021] Figure 5 An example method for performing processing using a configurable non-linear activation function circuit is depicted.

[0022] Figure 6 Depicted is an example of a pipelined sequential linear approximation architecture that can be used to implement nonlinear functions used in machine learning model acceleration.

[0023] Figure 7 Depicted is an example of a hyperbolic tangent (tanh) nonlinear activation function with non-uniform piecewise divisions for defining an approximate function output.

[0024] Figure 8 An example of a circuit for implementing a pipelined sequential linear approximation architecture is depicted.

[0025] Fig. 9 depicts an example of a linear approximator circuit, which may be Figure 8 Example of a linear approximator element in a circuit of FIG.

[0026] Fig.10 An example processing system is depicted that may be configured to perform the methods described herein.

[0027] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.

[0028] Detailed Description

[0029] Aspects of the present disclosure provide improved techniques for handling non-linear activation functions associated with machine learning models.

[0030] Nonlinear activation is a key component of various types of machine learning models, including neural network models. Although some nonlinear activation functions are implemented as piecewise linear functions (e.g., rectified linear units (ReLU), leaky ReLU, etc.), other nonlinear activation functions require complex mathematical functions (e.g., sigmoid, hyperbolic tangent (tanh), etc.). In some cases, complex mathematical functions can be implemented using interpolation (such as cubic spline interpolation). For example, in some aspects, a lookup table (LUT) can be used to determine the interpolated output value to match the output value to the input value. When a target input value is not mapped in the LUT, the LUT value associated with the input value adjacent to the target input value can be used to interpolate the output value for the target input value.

[0031] Conventionally, nonlinear activation functions can be implemented in software rather than hardware due to the wide range of possible activation functions available in machine learning models. However, such implementations typically require moving model data between processing devices (e.g., between a neural processing unit (NPU) that performs matrix multiplication and accumulation and a digital signal processor (DSP) that processes the nonlinear activation functions), thereby incurring power and latency penalties. Where nonlinear activation functions have been implemented in hardware, they are generally limited to supporting only a small number of nonlinear activation functions, and therefore cannot be configured to support evolving machine learning model architectures without falling back to outsourcing nonlinear activation function processing to distributed processing units.

[0032] For example, the rectified linear unit (ReLU) is a commonly used activation function in deep learning models. The function returns 0 if it receives a negative input, and returns the input x in other cases. Therefore, it can be written as f(x) = max(0, x). The ReLU function is generally not implemented by a main matrix multiplication and accumulation processing unit (such as a compute-in-memory (CIM) array in some examples). Therefore, from a processing point of view, the need to distribute the ReLU function or another nonlinear activation function is expensive. In addition, as the activation function becomes more complex, the processing cost also becomes more significant (e.g., for performing relatively high-power exponential and division operations as part of certain nonlinear activation functions, as described further below).

[0033] To overcome the shortcomings of conventional solutions, various aspects described herein relate to a configurable nonlinear activation (CNLA) function circuit that can be implemented in hardware for efficient processing. Specifically, because it can be implemented in hardware, the CNLA function can be co-located with other processing circuits optimized for other machine learning model processing tasks, such as CIM arrays and digital multiplication and accumulation (DMAC) circuits optimized for performing vector and matrix multiplication and accumulation functions.

[0034] In order to improve processing efficiency, various aspects described herein can use polynomial approximation to approximate complex functions, such as can be used in nonlinear activation functions. In some cases, various aspects described herein can use series expansion, such as Taylor series. In general, the Taylor series of a function (e.g., f (x)) is an infinite sum of terms represented by the derivative of the function at a single point. For many functions, the sum of the Taylor series of the function and the function is equal near the point. The partial sum formed by the first n+1 terms of the Taylor series is an n-order polynomial, which is referred to as the nth Taylor polynomial of the function. Therefore, Taylor polynomials allow efficient approximation of processing functions, which generally becomes better as n increases.

[0035] The CNLA function circuits described herein can implement one or more polynomial approximation blocks, such as a cubic approximation block, which generally enhances cubic spline interpolation to make it more efficient and more general, thereby covering a wider variety of nonlinear activation functions. In addition, the CNLA function circuits can be implemented as pipelined digital blocks that can use nonlinear piecewise lookup tables (LUTs) and approximations of mixed orders (e.g., pipelined linear, quadratic, and cubic approximations). Therefore, unlike conventional nonlinear activation function circuits, the CNLA function circuits described herein can be configured to meet many different performance goals.

[0036] Therefore, the CNLA function circuit described herein provides a technical solution to the technical problem of implementing a wide range of nonlinear activation functions in a machine learning model processing system. In addition, compared to existing solutions, the CNLA function circuit described herein provides technical improvements by increasing model processing performance, including lower latency, lower power consumption, improved memory efficiency, and other improvements as described herein.

[0037] Example configurable nonlinear activation function circuit

[0038] Figure 1 An example configurable non-linear activation (CNLA) function circuit 100 is depicted.

[0039] In general, the CNLA function circuit 100 may be configured to receive input data 101 (e.g., output values ​​from a layer of a machine learning model) and perform various non-linear activation functions to generate output data 114 (e.g., “activations”). The CNLA function circuit 100 may be co-located and pipelined with other machine learning model processing circuits (such as CIM arrays, DMACs, etc.), and may be configured to perform activation functions based on the outputs of the other machine learning model processing circuits.

[0040] In some examples, the input data 101 may be received from a buffer or other memory. In other examples, the input data 101 may be received directly from the output of another processing block, such as the output of a CIM array or another vector and matrix multiply and accumulate block.

[0041] The CNLA function circuit 100 includes a first approximator block 102, which may generally be configured to perform a hardware-based mathematical function (such as on input data 101). Figure 3 An example approximator is described in detail.

[0042] In some cases, the first approximator is a linear approximator (e.g., configured to perform a function such as ax+b), a quadratic approximator (e.g., configured to perform a function such as ax 2+bx+c) or a cubic approximator (e.g., configured to perform a function such as ax 3 +bx 2 +cx+d), where x is the input data and a, b, c, and d are configurable parameters. In general, a linear, quadratic, or cubic approximator can be used to approximate a given function, which may or may not be a polynomial function. The first approximator 102 can be configured with parameters retrieved from, for example, a memory, a register, a lookup table, etc. As described in further detail below with respect to Table 2, these different forms of approximation and associated configurable parameters can be used to approximate many types of nonlinear activation functions.

[0043] The CNLA function circuit 100 further includes a second approximator block 104, which, like the first approximator block 102, can generally be configured to perform hardware-based mathematical functions, such as linear, quadratic, or cubic functions. As described in more detail below, the CNLA function circuit 100 can be configured to use the first approximator block 102 and the second approximator block 104 in series to implement more complex functions, such that the output of the first approximator block 102 becomes the input to the second approximator block 104. The CNLA function circuit 100 can be further configured to use only one of the first approximator block 102 or the second approximator block 104 when processing simpler nonlinear functions, thereby saving power.

[0044] In some implementations, first approximator 102 and second approximator 104 may include the same circuit blocks (e.g., two instances of the same circuit element within circuit 100). For example, in some aspects, first approximator 102 and second approximator 104 may include cubic approximators. In other implementations, first approximator 102 and second approximator 104 may include different circuit elements, and in such cases, generally second approximator 104 will include a cubic approximator and first approximator 102 will include a lower order approximator, such as a quadratic or linear approximator. However, in other embodiments, the order of the higher order and lower order approximators may be reversed.

[0045] CNLA function circuit 100 includes a configurable bypass 105 that allows first approximator 102 to be bypassed in various scenarios, such as where the function only requires an approximator of a lower order than first approximator 102, and second approximator 104 is such a lower order approximator. For example, when first approximator 102 is bypassed via configurable bypass 105, then input data 101 is instead provided directly to second approximator 104 without being processed by first approximator 102. In various aspects, first approximator 102 may be a higher order approximator than second approximator 104, or vice versa, or they may be of the same order (e.g., linear, quadratic, or cubic). Configurable bypass 105 allows for saving processing time and energy when only one approximator is necessary.

[0046] The CNLA function circuit 100 further includes another configurable bypass 107 that allows the second approximator 104 to be bypassed in various scenarios, such as when the function only requires a first approximation (the first approximator 102 is capable of performing the function without the second approximator 104). For example, when the second approximator 104 is bypassed via the configurable bypass 107, the output of the first approximator 102 is directly provided to the multiplier 108.

[0047] In general, configurable bypasses 105 and 107 allow CNLA function circuit 100 to be configured for maximum versatility while saving power and avoiding unnecessary circuit block processing in various scenarios. In addition, configurable bypasses allow asymmetric and antisymmetric nonlinear activation functions to be configured for processing by CNLA function circuit 100. Figure 2 Example circuit aspects for implementing configurable bypasses 105 and 107 (eg, bypasses 205A and 205B) are depicted.

[0048] The CNLA function circuit 100 further includes a gain block 106 configured to provide a gain value to the multiplier 108. In some aspects, the gain block 106 is configured to generate a gain value 109 based on a gain function implemented by the gain block 106. In one example, the gain function may take the form of g=ax+b, where g is the gain value, x is the input data 101 value, and a and b are configurable parameters. More generally, the gain block 106 may multiplicatively (a) and / or additively (b) modify the input data to generate a gain value.

[0049] The gain value 109 generated by the gain block 106 is multiplied by the output of the first and / or second approximators 102 and 104 via the multiplier 108. In other aspects, the gain block 106 can be configured to generate a gain value that is not based on a function of the input data 101 (e.g., by setting a to zero in the above expression for g). In general, the parameters (e.g., a and b in the above example) or values ​​of the gain block 106 can be retrieved from, for example, a memory, a register, a lookup table, etc.

[0050] The CNLA function circuit 100 further includes a constant block 110 and an adder 112, wherein the constant block 110 is configured to store a configurable (e.g., programmable) constant value 113, and the adder 112 is configured to add the constant value 113 to the output of the multiplier 108 (e.g., a gain multiplier). The constant value 113 stored in the constant block 110 can be retrieved from, for example, a memory, a register, a lookup table, etc.

[0051] The inclusion and arrangement of the first approximator block 102, the second approximator block 104, the configurable bypasses 105 and 107, the gain block 106, the multiplier 108, the constant block 110, and the adder 112 allow the CNLA function circuit 100 to be configured to perform a wide variety of known and later developed nonlinear activation functions. In addition, the CNLA function circuit 100 can be efficiently configured to handle a wide variety of nonlinear activation functions by only updating the parameters for the first approximator 102, the second approximator 104, the gain block 106, and the constant block 110. When both approximator blocks 102 and 104 are used to simulate nonlinear functions, they can each be referred to as performing a separate function (e.g., a first function for the first approximator block 102 and a second function for the second approximator 104). This design beneficially supports arbitrary asymmetric nonlinear curves for complex functions.

[0052] Table 1 below provides the Figure 1 The CNLA function circuit 100 can be configured to perform various nonlinear activation functions, including example parameters for Figure 2 In Table 1, the gain is considered to have the form ax + b (as in Figure 1 ), but note that in other embodiments, the gain may be a scalar value or a different functional form. Similarly, the quadratic approximator is considered to have the form ax 2 +bx+c, and the cubic approximator is considered to have the form ax 3 +bx 2 +cx+d. In the following table, subscripts are used to indicate parameter assignments, for example, G for gain parameters, 1 for the first approximator, and 2 for the second approximator parameters.

[0053] Table 1

[0054]

[0055]

[0056]

[0057] Note that in the above ELU function, the α parameter is configurable as a hyperparameter by the model designer.

[0058] It is worth noting that in some implementations, the parameters for an approximator may be given in one form (e.g., cubic with a, b, c, and d parameters or quadratic with a, b, and c parameters) even where the approximator is performing a lower order function (e.g., linear). This is because, for example, setting the cubic parameter a to 0 effectively collapses the approximation equation to a lower order quadratic function, and likewise, setting the quadratic parameter a to 0 effectively collapses the approximation equation to a linear equation. Thus, when an approximator is configured with quadratic parameters, the approximator may be configured, for example, for a "quadratic function," but the result of the parameters may simplify the function to a linear function, as in the example of ReLU in Table 2 above. This may allow standardization of parameter sets regardless of the order of the underlying function to be configured by the parameters, thereby simplifying implementation.

[0059] Figure 2 Depicted are example circuit blocks 202 and 204 for implementing bypassable approximator blocks 206A and 206B. In one example, bypassable approximator blocks 206A and 206B may correspond to Figure 1 The first approximator block 102 and the second approximator block 104 are described in detail.

[0060] exist Figure 2 , circuit block 202 is configured to control the use of function block 214A, which in this example includes a first approximator 206A and a minimum and maximum function block 208A. Similarly, circuit block 204 controls the use of function block 214B, which in this example includes a minimum and maximum function block 208B and a second approximator 206B. The first and second approximator blocks 206A and 206B may be configured to implement non-linear activation functions, such as those described above with respect to Table 1.

[0061] Notice, Figure 1The first approximator 102 in FIG. 1 requires only one input, but the circuit block 202 includes two input ports 201A and 201B that allow for multiple inputs. The depicted configuration of the circuit block 202 can be employed to provide the same external interface to both circuit blocks 202 and 204, which can simplify configuration and integration. In some aspects, the two input ports 201A and 201B of the circuit block 202 receive a single input (such as 101A) via the input port 201A of the circuit block 202. Figure 1 In an alternative implementation, circuit block 202 may be simplified by removing input port 201B and removing input multiplexer 203A so that 201A will be directly provided to 214A and 207A.

[0062] Generally speaking, input ports 201A and 201B can receive various types of input data (including signed multi-bit integer data) for processing. In one example, the input data is 8-bit two's complement input data.

[0063] Input selector multiplexers 203A and 203B are configured to respectively control which input data port is used for circuit blocks 202 and 204. For example, input selector multiplexer 203B may select between input data port 201A (e.g., when circuit block 202 is bypassed) or 212B (e.g., when circuit blocks 202 and 204 are processed in series).

[0064] Bypass selector multiplexers 211A and 211B are configured to control bypass function blocks 214A and 214B of circuit blocks 202 and 204, respectively. For example, when circuit block 202 is to be bypassed, bypass selector multiplexer 211A selects bypass line 205A to provide an output to output port 212A. Similarly, when circuit block 204 is to be bypassed, bypass selector multiplexer 211B selects bypass line 205B to provide an output to output port 216. Thus, processing utilizing circuit blocks 202 and / or 204 results in an output at output port 216, as controlled by configurable bypasses 205A and 205B.

[0065] As about Figure 3 As discussed in more detail, approximator blocks 206A and 206B may be configured with configuration parameters (e.g., function-specific coefficients as in Table 1 above) stored in registers 219A and 219B, respectively. Similarly, as in Table 1 above, where approximator blocks 206A or 206B are configured to perform a lookup table-based function, the table values ​​may be stored in registers 219A and 219B, respectively.

[0066] Each circuit block (202 and 204) further includes a minimum and maximum function block (208A for circuit block 202 and 208B for circuit block 204) for providing minimum and maximum functions. Generally speaking, a minimum (or "min") function will return the minimum value of the input provided. Similarly, a maximum (or "max") function will return the maximum value of the input provided. In one example, the minimum and maximum function blocks 208A and 208B may include multi-bit digital comparators operating in a single-cycle or multi-cycle mode.

[0067] The configuration of function blocks 214A and 214B may include settings for function selector multiplexers 209A and 209B, respectively. In other words, whether function blocks 214A and 214B output the min / max outputs from min / max blocks 208A and 208B or the values ​​from approximators 206A and 206B is based on the configuration of function selector multiplexers 209A and 209B. Note that in other examples, function blocks 214A and 214B may include additional function blocks that may be selected by the multiplexers.

[0068] As in the case where the approximator blocks can be processed in series Figure 1 As depicted in Figure 2 , the output 212A of the circuit block 202A including the first approximator block 206A is provided as an input 212B to the circuit block 204 including the second approximator block 206B. Figure 1 In this embodiment, the use of bypass 205A and 205B control approximator blocks 206A and 206B can be selected.

[0069] Asymmetric signal line 210 controls the configuration of circuit blocks 202 and 204. In one example, circuit blocks 202 and 204 are configured based on the value on asymmetric signal line 210 and the output values ​​from sign blocks 207A and 207B, which are based on input data received via input data port 201A. For example, the binary value received via asymmetric signal line 210 and the binary value output from sign block 207A interact at AND gate 213 to control the selection of the output of multiplexer 211A. As another example, the binary value received via asymmetric signal line 210 and the binary value output from sign block 207B interact at AND gate 217 to control the selection of the input data port (between 201A and 212B) via multiplexer 203B. As a further example, the binary value received via the asymmetric signal line 210 and the inverted binary value output from the sign block 207B interact at the AND gate 215 to control the selection of the output of the multiplexer 21 IB.

[0070] Table 2 below provides a summary of the configuration for circuit blocks 202 and 204:

[0071] Table 2

[0072]

[0073] Example approximator for a configurable nonlinear activation function circuit

[0074] Figure 3 An example approximator 300 is depicted, which may be Figure 1 The first approximator 102 and the second approximator 104 and / or Figure 2 An example of one or both of approximators 206A and 206B.

[0075] Approximator 300 receives input data 302 (e.g., pre-activation data) for processing. In some examples, input data 302 may be received from a buffer or other memory. In other examples, input data may be received directly from the output of another processing block (such as a CIM array or the output of another vector and matrix multiplication and accumulation block). In addition, input data may be received from another approximator (such as when approximator 300 is Figure 1 The second approximator 104 in Figure 2 206B in the case of the second approximator 206B).

[0076] In some implementations, an approximator such as 300 may include alternate processing paths. In such cases, path logic 304 may be configured to route input data 302 to the appropriate processing path based on, for example, configuration parameters for approximator 300.

[0077] In this example, processing path 306A provides a cubic approximation path for input data 302 .

[0078] In processing path 306A, input data 302 is provided to a cubic calculator 308, which performs a cubic operation (eg, x 3 , where x is the input data), and the output is then multiplied at multiplier 310 with a cubic parameter 312. The output of multiplier 310 is then provided to accumulator 324.

[0079] The input data 302 is also provided to a quadratic calculator 308, which performs a quadratic operation (eg, x 2 , where x is the input data), and the output is then multiplied by a quadratic parameter 318 at a multiplier 316. The output of the multiplier 316 is then provided to an accumulator 324.

[0080] The input data 302 is also provided to a multiplier 320 where the input data 302 is multiplied by a linear parameter 322. The output of the multiplier 320 is then provided to an accumulator 324.

[0081] An accumulator (adder) 324 accumulates the outputs of the multipliers 310 , 316 , and 320 and an intercept parameter 326 to generate output data 332 .

[0082] The cubic parameters 312, quadratic parameters 318, linear parameters 322, and intercept parameters 326 may all be stored in a memory accessible to the accumulator 300, etc. (e.g., in registers). In some cases, a control unit (such as a memory control unit or a finite state machine) may configure the approximator 300 using the parameters stored in the memory. In various examples, the cubic parameters 312, quadratic parameters 318, linear parameters 322, and intercept parameters 326 may be set according to the values ​​described above with respect to Table 2.

[0083] As described above, the order of the approximation can be configured by configuring the above-mentioned parameter values. For example, in order for the approximator 300 to perform a quadratic approximation, the cubic parameter 312 can be set to zero. Similarly, in order for the approximator 300 to perform a linear approximation, the cubic parameter 312 and the quadratic parameter 318 can be set to zero.

[0084] Certain non-linear activation functions require replacement functions, such as minimum and maximum functions. Therefore, processing path 306B provides a minimum and / or maximum calculator that can be used, for example, with the ReLU and ReLU6 functions described above in Table 2. Path logic 304 can select processing path 306B based on configuration data for approximator 300.

[0085] In addition, certain non-linear activation functions may be implemented using lookup tables, which provide a more power and time efficient mechanism for generating values ​​for certain non-linear activation functions. Thus, processing path 306C provides a lookup table based processing path that may be used, for example, in any case where the non-linear activation function uses a sigmoid, tanh, or similar function. Note that sigmoid and tanh may be calculated from each other, so in some cases only a single lookup table (e.g., either sigmoid or tanh, but not both) is stored and used to implement both functions. One or more lookup tables may be stored in a memory and accessible to the approximator 300, including a memory tightly coupled to the approximator 300.

[0086] Example machine learning model process flow using configurable nonlinear activation function circuits

[0087] Figure 4 Describes the implementation of the above Figures 1 to 3 An example machine learning model data flow 400 of a configurable non-linear activation function circuit is described.

[0088] In process 400, input data is stored in input data buffer 401 (e.g., machine learning model layer input data) and is then provided to multiplication and accumulation (MAC) circuit 402. MAC circuit 402 can generally be configured to perform vector, array, and matrix multiplication and accumulation operations, such as those frequently used in convolutional neural networks. In some examples, MAC circuit 402 may include one or more computation in memory (CIM) arrays. Alternatively or additionally, MAC circuit 402 may include digital multiplication and accumulation (DMAC). In yet other examples, multiplication and accumulation circuit 402 may be part of a machine learning accelerator, such as a neural processing unit (NPU), or another type of processing unit optimized for performing machine learning processing. In another implementation, MAC circuit 402 may be replaced by a vector / matrix or matrix / matrix processing engine.

[0089] MAC circuit 402 processes input data using weight data (e.g., neural network weight data) to generate pre-activation data. For example, MAC circuit 402 can process input data to a neural network model layer and generate pre-activation data as output.

[0090] The pre-activation data is provided to a configurable non-linear activation (CNLA) function circuit 404, which is configured to generate output data (e.g., activation) based on the configured non-linear activation function. The output data may then be stored in an output data buffer 405 for subsequent use, such as for processing another layer in a machine learning model, or as output from a machine learning model, etc.

[0091] The CNLA function circuit 404 can utilize configuration parameters (such as Figure 3 4 and those described in Tables 1 and 2). In addition, the CNLA function circuit 404 may be configured to access a lookup table depending on the configured activation function.

[0092] In some cases, the configuration parameters may include an identification of a nonlinear activation function to be applied to the input data. Based on the determined nonlinear activation function, appropriate parameters (such as those in Table 2) may be retrieved from a memory (e.g., a register) and applied to the CNLA function circuit 404, thereby configuring it to process the input data. In some examples, a finite state machine, a memory control unit, or another controller may perform the configuration of the CNLA function circuit 404.

[0093] It is worth noting that the CNLA circuit 404 can be configured to process multiple batches of input data using the same configuration, or its configuration can be updated for each new batch of input data. Therefore, the CNLA circuit 404 provides a very flexible and efficient means for performing configurable nonlinear activations for machine learning tasks such as training and inference.

[0094] Example method for performing processing using a configurable non-linear activation function circuit

[0095] Figure 5 An example method 500 for performing processing using a configurable non-linear activation function circuit is depicted.

[0096] The method 500 begins at step 502 by selecting a nonlinear activation function for application to input data. For example, the nonlinear activation function may be one of the functions listed in Table 2, or another nonlinear activation function.

[0097] The method 500 then proceeds to step 504 to determine a parameter set for a configurable nonlinear activation function circuit based on the selected nonlinear activation function. For example, the parameters for the determined nonlinear activation function may be as shown in Tables 1 and 2 above.

[0098] The method 500 then proceeds to step 506 to process the input data using a configurable nonlinear activation function circuit based on the parameter set to generate output data. For example, the output data may be activation data for a layer of a neural network model.

[0099] In some examples, the parameter set includes a combination of one or more gain parameters, a constant parameter, and one or more approximation functions to be applied to the input data via a configurable nonlinear activation function circuit. For example, the parameter set may be as described above with respect to Figure 1 and Figure 2 and those discussed in Table 1.

[0100] In some examples, method 500 further includes retrieving a parameter set from a memory based on the determined non-linear activation function. In some examples, the memory may be one or more registers storing parameter values.

[0101] In some examples, the configurable nonlinear activation function circuit includes a first approximator configured to approximate a first function of the one or more approximation functions; a second approximator configured to approximate a second function of the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value, such as about Figure 1 Depicted and described.

[0102] In some examples, the configurable non-linear activation function circuit includes a first bypass configured to bypass the first approximator. In some examples, the configurable non-linear activation function circuit includes a second bypass configured to bypass the second approximator. In some examples, the configurable non-linear activation function circuit includes an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0103] In some examples, at least one of the first approximator and the second approximator is a cubic approximator. In some examples, the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator. In some examples, the other of the first approximator and the second approximator is configured to perform a minimum or maximum function, such as a function of Figure 3 In some examples, the other of the first approximator and the second approximator is configured to access a lookup table to find an approximation value, such as Figure 3 As depicted by path 306C in FIG.

[0104] In some examples, both the first approximator and the second approximator are cubic approximators.

[0105] Notice, Figure 5 This is merely one example, and in other examples, methods such as those described herein may be implemented with more, fewer, and / or different steps.

[0106] Example pipelined approximation for nonlinear functions

[0107] Figure 6 An example of a pipelined sequential linear approximation architecture 600 that can be used to implement nonlinear functions used in machine learning model acceleration is depicted. For example, the architecture 600 can be used to implement Figure 1 The approximators 102 and 104 described and the Figure 2 206A and 206B as described.

[0108] Generally speaking, Figure 6 The input x is depicted flowing through the linear approximators 604A-C in a pipelined manner. The architecture 600 is a power and space efficient way to implement cubic approximation using sequential linear approximation blocks 604A-C. Specifically, the input x is processed in the linear approximator block 604A to determine the output y 1 =Ax+B, where A and B are the coefficients 605A used by the linear approximator block 604A.

[0109] Next, y 1 is provided as input to the linear approximator block 604B to determine the output y 2 =x(y 1)+C, where C is the coefficient 605B used by the linear approximator block 604B. Note that x(y 1 ) indicates y 1 Multiply by x.

[0110] Next, y 2 is provided as input to the linear approximator block 604C to determine the output y 3 =x(y 2 )+D, where D is the coefficient 605C used by the linear approximator block 604C. 3 gives:

[0111] y 3 =x(y 2 )+D

[0112] =x(x(Ax+B)+C)+D

[0113] =Ax 3 +Bx 2 +Cx+D,

[0114] It is based on a cubic approximation of the input x. Notably, the successive linear approximation approach implemented in architecture 600 reduces the number of multipliers required, which reduces complexity and power used compared to conventional cubic approximator architectures. Furthermore, architecture 600 allows for a pipelined implementation that produces a new output every cycle in the steady state, and thus improves throughput.

[0115] To further improve processing efficiency, the coefficients A, B, C, and D (605A-C) may be stored in registers rather than, for example, SRAM or DRAM, which improves the power and latency performance of the architecture 600. For example, by using local registers to store the coefficient values, the use of read power from the main memory system SRAM may be avoided. The register approach also supports pipelined architectures by avoiding additional cycles to read and transfer coefficient values ​​from a remote main memory system memory (e.g., DRAM). In various aspects, the pipeline registers may be internal or external to the linear approximators 604A-C.

[0116] In some implementations, as depicted with respect to architecture 600, coefficients may be determined based on an approximation of a nonlinear function of a non-uniform segmentation, as described with respect to Figure 7 In such an embodiment, the region finder 602 can determine appropriate coefficients based on the region (or segment) of the non-uniform piecewise approximation of the input x. Therefore, Figure 6 Depicted is the output of the region finder "REGION_SEL" passed through the various stages of linear approximation in a pipelined manner.

[0117] Note that the "regions" found by region finder 602 are segments of the "domain" of the original nonlinear function. That is, a nonlinear function can be defined over a domain from negative infinity to positive infinity, and the domain can be divided into segments. In various aspects described herein, the segments are of non-uniform length. For each such segment, coefficients are selected to best approximate the function within the segment. Figure 8 An example circuit implementation is depicted with a region finder element 809 where "cutoff" is used to describe a segment endpoint.

[0118] In various aspects, region finder 602 can be configured to select an entire set of coefficients for the entire approximation. In the depicted example, because the approximation occurs serially in a pipelined manner, the determined regions will be registered as running along each approximator data path, and thus the determined regions can be used for subsequent coefficient lookups at each stage (e.g., for linear approximators 604A-C).

[0119] The region finder 602 generally allows the use of various offline algorithms to search and set the region selection, so the implementation need not be limited by how the regions are found.

[0120] For example, a cubic spline algorithm can be used to ensure continuous derivatives across region boundaries, which is beneficial for model training. In another example, the mean square error can be minimized on a per-point basis (based on a subsequent search for the best cubic coefficients given the region being considered). This second approach can achieve better computational accuracy (achieving better inference accuracy) at the expense of continuous derivatives. Yet another approach is to set the region based on the second or third order derivative being 0, although this approach may not use the full set of possible regions because more regions result in less error. Finally, architecture 600 is configured to exploit the even-odd symmetry properties of common neural network nonlinear functions (such as sigmoid and hyperbolic tangent (tanh)) to beneficially reduce circuit area and power during processing. Beneficially, exploiting such symmetry allows a reduction in the number of coefficients that need to be stored to approximate a nonlinear function, because, for example, a simple sign flip can simulate both sides of the function (e.g., positive and negative) from the origin without the need to store coefficients for both sides. In some implementations, approximate results are calculated only for positive inputs, and then the same result may be used for negative inputs with the same magnitude (even symmetry), or in the case of functions with odd symmetry, the sign of the result may be flipped.

[0121] In the depicted example, the sign and offset corrector component 606 is configured to convert the cubic approximation y 3, the original input x (e.g., to determine its sign), and the offset value 607 are taken as inputs, and sign and / or offset correction is applied to generate the final output 608. For example, the final output 608 can be calculated as the offset (607) + / - y 3 As with coefficients 605A-C, offset 607 may be stored in a local register for efficiency and speed.

[0122] In general, the offset 607 can be used for symmetric functions except those not centered at x = 0. An example is the sigmoid, which is symmetric about the line y = 0.5. In such a case, a "shifted" sigmoid is first approximated with odd symmetry about the x-axis, and then an offset of 0.5 is added to the result.

[0123] Notice, Figure 6 An example architecture 600 is depicted that includes three linear approximators (604A-C) that can be used to approximate a cubic nonlinear function. In other aspects, where the goal is to approximate a quadratic nonlinear function, two linear approximators can be used. Figure 6 , but in some aspects, when only a quadratic function needs to be approximated, the control logic may switch the output of a linear approximator (such as linear approximator 604B) to bypass a subsequent linear approximator (such as linear approximator 604C). In such a case, the bypass may obtain the output y of linear approximator 604B. 3 and routed to the symbol and offset corrector component 606.

[0124] Figure 7 An example 700 of a hyperbolic tangent (tanh) nonlinear activation function with non-uniform segmentation for defining an approximation function output is depicted.

[0125] In the depicted example, a first segment 702A is defined along the curve 701 from the origin (x=0) to a first point. As depicted, segments such as 702B are longer along the length of the curve 701, where the slope is more constant across the input values ​​in the segment, while segments such as 702C are shorter along the length of the curve 701, where the curve changes slope more quickly along the input values ​​in the segment. Using non-uniform segmentation beneficially allows fewer segments to be defined across the output value range of the non-linear function, while still maintaining a similar output resolution compared to the actual function. Conventionally, in order to maintain resolution, curves (such as 701) are segmented uniformly, which means that many segments have very similar output values ​​and therefore produce redundant values ​​stored in memory.

[0126] Note that each segment can generally be defined by a start point and an end point along the curve 701. For example, the starting input value x 开始 and the end input value x 结束A segment can be defined. Each of these segments can be used to approximate the input value (such as x) based on the hyperbolic tangent function that falls within the defined range. 开始 ≤x <x 结束 ). To further save memory, the first segment 702A only needs to store the end point of the segment, because the starting point (origin) can be assumed to be x=0. Similarly, the last segment is defined only by the end point of the second-to-last segment. That is, all inputs greater than or equal to the end point of the second-to-last segment are considered part of the last segment.

[0127] For example, the following table gives an example of non-uniform segments that can be stored in registers for a tanh function to be approximated. When an input value x falls within a particular segment (or region), the coefficients for that segment can be retrieved and used to generate an approximated function value.

[0128] Table 3

[0129] Segmentation <![CDATA[x 开始 ]]> <![CDATA[x 结束 ]]> 1 (Assumed to be 0) 0.1250 2 0.1250 0.3750 3 0.3750 0.6250 4 0.6250 0.8750 5 0.8750 1.3750

[0130] Note that the hyperbolic tangent function curve 701 is an example of a function with odd symmetry. That is, the tanh output value for a positive input (e.g., 0.2 for point 704A) is equivalent to the sign-flipped output for a negative input of the same magnitude (e.g., -0.2 for 704B). Therefore, the non-uniform segmentation only needs to be defined with respect to the range of positive input values ​​of the hyperbolic tangent function curve 701, and the sign corrector component (such as Figure 6 606 in ) can appropriately flip the final output (e.g., Figure 6 608) symbol.

[0131] Figure 8 Describes a method for implementing the above Figure 6 An example of a pipelined sequential linear approximation architecture circuit 800 is described. Specifically, Figure 8 The pipelined nature of the design is depicted, including how the output of the region finder 809 and the sign bit are pipelined for efficient processing.

[0132] The depicted example includes four stages 806A-806D, three of which include linear approximators (804A-804C, respectively) and one sign and offset correction stage 806D.

[0133] In the depicted example, various operation blocks are depicted. "SAT" represents a "saturation" block. Saturation is useful when reducing the number of integer bits used to represent a value. If the saturation module is used to reduce an input with M integer bits to an output with N integer bits (where N < M), input values greater than what can be represented with N integer bits are "saturated" to the largest positive N-bit number (for positive inputs) or the smallest negative N-bit number (for negative inputs). For example, if [x > 0111...11], then the output y(0111...11), or if [x <= 1000...00], then the output y(1000...00).

[0134] "ABS" represents a block that calculates the absolute value (or magnitude) of the input.

[0135] "COMP" represents a comparator block that functions as a region finder with a programmable cutoff. The cutoff defines the end points of the individual segments (and indirectly, also the start point of the last segment). The output of the COMP block 809 is an identifier indicating which region (segment) the current input belongs to. This identifier is used to select the appropriate set of coefficients to approximate the nonlinear function within the segment.

[0136] In one example implementation, 11 cutoff values are used to define a total of 12 segments. By exploiting the symmetric property of the nonlinear function, the COMP module operates only on the absolute value of the input. The following algorithm describes its operation:

[0137] If 0 <= ABS(x) < CUTOFF1: Select segment 0 coefficients

[0138] Else if (CUTOFF1 <= ABS(x) < CUTOFF2): Select segment 1 coefficients

[0139] Else if (CUTOFF2 <= ABS(x) < CUTOFF3): Select segment 2 coefficients

[0140] …

[0141] Else if (CUTOFF10 <= ABS(x) < CUTOFF11): Select segment 10 coefficients

[0142] Else if (CUTOFF11 <= ABS(x)): Select segment 11 coefficients

[0143] "TRUNC" represents a truncation block that is configured to reduce the precision of a fraction represented in two's complement form by omitting the least significant bits. For example, to reduce an input from having M fraction bits to having N fraction bits, the least significant M - N bits are omitted (truncated).

[0144] "SYMMSAT" stands for "Symmetrical Saturation" block. A conventional saturation (SAT) block with N integer bits saturates to the maximum positive value (2 N-1 -1) or the smallest negative value (-2 N-1 In the symmetric saturation block, negative inputs are instead saturated to a value greater than the minimum negative value (e.g. (-2 N-1 +1)). In this way, the number of different negative and positive outputs is the same. For example, if [x>0111...11], then y(0111...11) is output, or if [x<=1000...00], then y(1000..0.01) is output.

[0145] "SE" stands for Sign Extension Block, which is useful when increasing the number of integer bits used to represent a value. For example, when a value represented by N integer bits is required to be represented by M integer bits (M>N), the sign bit is replicated (MN) times.

[0146] "TC" indicates a block for converting a positive number to a negative number by taking its two's complement.

[0147] Finally, "ZF" refers to the "zero fill" block, which is used when a number with N fractional bits is required to be represented as a number with M fractional bits (M>N) by appending (MN) zeros in the least significant positions.

[0148] Fig. 9 An example of a linear approximator circuit 900 is depicted, which may be Figure 8 800. As depicted, the linear approximator circuit includes a multiplication element 902 that takes as inputs a stage input and a coefficient 904, and an adder element 906 that takes as inputs the output of the multiplication element after processing by the truncation and saturation blocks and a second coefficient 908.

[0149] Example Processing System

[0150] Fig.10 Describes a method that can be configured to perform the Figure 4-5 An example processing system 1000 of the described methods.

[0151] Processing system 1000 includes a central processing unit (CPU) 1002, which in some examples may be a multi-core CPU. Instructions executed at CPU 1002 may be loaded, for example, from a program memory associated with CPU 1002 or may be loaded from memory partition 1024.

[0152] The processing system 1000 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 1004 , a digital signal processor (DSP) 1006 , a neural processing unit (NPU) 1008 , a multimedia processing unit 1010 , and a wireless connectivity component 1012 .

[0153] An NPU, such as 1008, is generally a dedicated circuit configured to implement all necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. An NPU is sometimes referred to interchangeably as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0154] NPUs such as 1008 may be configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs may be instantiated on a single chip such as a system on a chip (SoC), while in other examples, multiple NPUs may be part of a dedicated machine learning accelerator device.

[0155] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0156] NPUs configured to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters (such as weights and biases) to improve model performance. Generally speaking, optimization based on error prediction involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0157] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs can thus be configured to input new data segments and quickly process them through the trained model to generate model output (e.g., inference).

[0158] In some embodiments, NPU 1008 may be implemented as part of one or more of CPU 1002 , GPU 1004 , and / or DSP 1006 .

[0159] In some embodiments, the wireless connectivity component 1012 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 1012 is further connected to one or more antennas 1014.

[0160] The processing system 1000 may also include one or more sensor processing units 1016 associated with any manner of sensors, one or more image signal processors (ISPs) 1018 associated with any manner of image sensors, and / or a navigation processor 1020, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0161] The processing system 1000 may also include one or more input and / or output devices 1022, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0162] In some examples, one or more processors of processing system 1000 may be based on the ARM or RISC-V instruction set.

[0163] Processing system 1000 also includes various circuits according to various embodiments described herein.

[0164] In this example, the processing system 1000 includes a compute in memory (CIM) circuit 1026, which can be configured to perform an efficient multiplication and accumulation (MAC) function for processing machine learning model data. The processing system 1000 further includes a configurable nonlinear activation (CNLA) function circuit 1028. In some cases, the CNLA function circuit 1028 can be similar to the one described with respect to Figure 2 , 3 , 6, 8 and 9. The CNLA function circuit 1028 and other circuits not depicted may be configured to perform the methods described herein (such as respectively Figure 4 and Figure 5 Various aspects of methods 400 and 500).

[0165] In some examples, CNLA function circuit 1028 may be implemented as part of another processing unit, such as CPU 1002 , GPU 1004 , DSP 1006 , or NPU 1008 .

[0166] The processing system 1000 also includes a memory 1024, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 1024 includes computer-executable components that can be executed by one or more of the aforementioned components of the processing system 1000.

[0167] Specifically, in this example, memory 1024 includes a determination component 1024A, a configuration component 1024B, a processing component 1024C, a retrieval component 1024D, non-linear activation function parameters 1024E, (all) lookup tables 1024F, and model parameters 1024G (e.g., weights, biases, and other machine learning model parameters). One or more of the depicted components and other components not depicted may be configured to perform various aspects of the methods described herein.

[0168] In general, the processing system 1000 and / or its components may be configured to perform the methods described herein.

[0169] It is noted that in other embodiments, aspects of the processing system 1000 may be omitted, such as where the processing system 1000 is a server computer, etc. For example, in other embodiments, the multimedia component 1010, the wireless connectivity 1012, the sensor 1016, the ISP 1018, and / or the navigation component 1020 may be omitted. Furthermore, aspects of the processing system 1000 may be distributed.

[0170] Notice, Fig.10 This is merely one example, and alternative processing systems having more, fewer, and / or different components may be used in other examples.

[0171] Sample Clauses

[0172] Implementation examples are described in the following numbered clauses:

[0173] Item 1: A processor comprising: a configurable non-linear activation function circuit, the configurable non-linear activation function circuit being configured to: determine a non-linear activation function for application to input data; determine a parameter set for the non-linear activation function based on the determined non-linear activation function; and generate output data based on application of the parameter set for the non-linear activation function.

[0174] Item 2: A processor as in Item 1, wherein the configurable nonlinear activation function circuit comprises: a first approximator configured to approximate a first function using one or more first function parameters of the parameter set; a second approximator configured to approximate a second function using one or more second function parameters in the parameter set; a gain multiplier configured to multiply a gain value based on one or more gain parameters in the parameter set; and a constant adder configured to add a constant value based on a constant parameter in the parameter set.

[0175] Clause 3: The processor of clause 2, wherein at least one of the first approximator and the second approximator is a cubic approximator.

[0176] Clause 4: The processor of clause 3, wherein the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator.

[0177] Clause 5: The processor of clause 2, wherein both the first approximator and the second approximator are cubic approximators.

[0178] Clause 6: The processor of clause 3, wherein the other of the first approximator and the second approximator is configured to access a lookup table to find the approximate value.

[0179] Clause 7: The processor of clause 3, wherein the other of the first approximator and the second approximator is configured to perform a minimum or maximum function.

[0180] Clause 8: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a swish function, the gain parameter comprises a dependent parameter value of 1 and an independent parameter value of 0, the constant value is 0, the first function is a quadratic function, and the second function is a sigmoid lookup table.

[0181] Clause 9: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a hard swish function, the gain parameter comprises a dependent parameter value of 1 / 6 and an independent parameter value of 0, the constant value is 3, the first function is a max function, and the second function is a min function.

[0182] Clause 10: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a hyperbolic tangent (tanh) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a tanh lookup table.

[0183] Clause 11: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a sigmoid function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a linear function, and the second function is a sigmoid lookup table.

[0184] Clause 12: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a Gaussian error linear unit (GELU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 1, the first function is a cubic function, and the second function is a tanh lookup table.

[0185] Clause 13: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a rectified linear unit (ReLU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a max function.

[0186] Clause 14: A processor as in clause 2, wherein: the determined nonlinear activation function comprises a rectified linear unit (ReLU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a max function, and the second function is a min function.

[0187] Item 15: A processor as in Item 2, wherein: the determined nonlinear activation function comprises an exponential linear unit (ELU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of α, the constant value is 0, the first function is a quadratic function when the input data value is ≥0; or is bypassed when the input data value is <0; the second function is bypassed when the input data value is ≥0; or is an exponential lookup table when the input data value is <0.

[0188] Clause 16: A processor as in any of clauses 1-15, further comprising: an input memory buffer configured to store one or more outputs received from the processing circuit as input data; and an output memory buffer configured to store generated output data for output from the configurable nonlinear activation function circuit.

[0189] Clause 17: The processor of any of clauses 1-16, further comprising: an in-memory computation array configured to provide input data to the configurable non-linear activation function circuit.

[0190] Item 18: A method for processing input data by a configurable nonlinear activation function circuit, comprising: determining a nonlinear activation function for application to the input data; determining a parameter set for the configurable nonlinear activation function circuit based on the determined nonlinear activation function; and processing the input data using the configurable nonlinear activation function circuit based on the parameter set to generate output data.

[0191] Clause 19: The method of clause 18, further comprising: retrieving a parameter set from a memory based on the determined non-linear activation function.

[0192] Clause 20: The method of clause 18, wherein the parameter set comprises a combination of one or more gain parameters, a constant parameter, and one or more approximation functions to be applied to the input data via a configurable non-linear activation function circuit.

[0193] Item 21: A method as in Item 20, wherein the configurable nonlinear activation function circuit comprises: a first approximator configured to approximate a first function among the one or more approximation functions; a second approximator configured to approximate a second function among the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value based on a constant parameter.

[0194] Clause 22: A method as in clause 21, wherein the configurable nonlinear activation function circuit further comprises: a first bypass configured to bypass the first approximator; a second bypass configured to bypass the second approximator; and an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0195] Clause 23: The method of clause 22, wherein at least one of the first approximator and the second approximator is a cubic approximator.

[0196] Clause 24: The method of clause 23, wherein the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator.

[0197] Clause 25: The method of clause 23, wherein both the first approximator and the second approximator are cubic approximators.

[0198] Clause 26: The method of clause 23, wherein the other of the first approximator and the second approximator is configured to access a lookup table to find the approximate value.

[0199] Clause 27: The method of clause 23, wherein the other of the first approximator and the second approximator is configured to perform a min or max function.

[0200] Clause 28: A method as in clause 21, wherein: the determined nonlinear activation function comprises a swish function, the gain parameter comprises a dependent parameter value of 1 and an independent parameter value of 0, the constant value is 0, the first function is a quadratic function, and the second function is a sigmoid lookup table.

[0201] Clause 29: A method as in clause 21, wherein: the determined nonlinear activation function comprises a hard swish function, the gain parameter comprises a dependent parameter value of 1 / 6 and an independent parameter value of 0, the constant value is 3, the first function is a max function, and the second function is a min function.

[0202] Clause 30: A method as in Clause 21, wherein: the determined nonlinear activation function includes a Gaussian error linear unit (GELU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 1, the first function is a cubic function, and the second function is a tanh lookup table.

[0203] Clause 31: A method as in Clause 21, wherein: the determined nonlinear activation function includes a hyperbolic tangent (tanh) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a tanh lookup table.

[0204] Clause 32: A method as in Clause 21, wherein: the determined nonlinear activation function comprises a sigmoid function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a linear function, and the second function is a sigmoid lookup table.

[0205] Clause 33: A method as in Clause 21, wherein: the determined nonlinear activation function includes a rectified linear unit (ReLU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a max function.

[0206] Clause 34: A method as in Clause 21, wherein: the determined nonlinear activation function includes a rectified linear unit (ReLU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a max function, and the second function is a min function.

[0207] Clause 35: A method as in Clause 21, wherein: the determined nonlinear activation function comprises an exponential linear unit (ELU) function, the gain parameter comprises a dependent parameter value of 0 and an independent parameter value of α, the constant value is 0, the first function is a quadratic function when the input data value is ≥0; or is bypassed when the input data value is <0; the second function is bypassed when the input data value is ≥0; or is an exponential lookup table when the input data value is <0.

[0208] Clause 36: The method of clause 21, further comprising: receiving input data from a compute-in-memory (CIM) array.

[0209] Item 37: A configurable non-linear activation function circuit configured to process a non-linear activation function according to any configuration of Table 1.

[0210] Item 38: A circuit block comprising: a configurable nonlinear activation function circuit; and a selectable bypass.

[0211] Clause 39: The circuit block of clause 38, wherein the configurable non-linear activation function circuit is configured to process the non-linear activation function according to any configuration of Table 1.

[0212] Clause 40: The circuit block of clause 38, wherein the circuit block is configurable according to any configuration of Table 2.

[0213] Item 41: A processor, comprising: a configurable nonlinear activation function circuit, the configurable nonlinear activation function circuit being configured to: determine a parameter set for a selected nonlinear activation function based on a selected nonlinear activation function; and generate output data based on application of the parameter set for the selected nonlinear activation function, wherein: the configurable nonlinear activation function circuit includes at least one nonlinear approximator, the at least one nonlinear approximator includes at least two consecutive linear approximators, and each linear approximator of the at least two consecutive linear approximators is configured to approximate a linear function using one or more function parameters in the parameter set.

[0214] Clause 42: The processor of clause 41, wherein each linear approximator of the at least two successive linear approximators comprises: a stage input; a coefficient input; a constant input; and a stage output.

[0215] Clause 43: The processor of any of clauses 41-42, wherein the at least one non-linear approximator comprises a cubic approximator having three successive linear approximators.

[0216] Clause 44: The processor of any of Clauses 41-43, wherein the at least one nonlinear approximator comprises a quadratic approximator having two successive linear approximators.

[0217] Clause 45: A processor as in clause 42, wherein: the at least one nonlinear approximator further comprises a region finder component configured to determine an input value region, and each of the at least two successive linear approximators is further configured to determine a coefficient input and a constant input based on the input value region.

[0218] Clause 46: The processor of any of clauses 41-45, wherein the at least one nonlinear approximator further comprises a sign and offset corrector component configured to modify a stage output from at least one of the at least two consecutive linear approximators.

[0219] Clause 47: The processor of clause 46, wherein the sign and offset corrector component is further configured to invert the sign of the stage output so as to modify the stage output from at least one of the at least two successive linear approximators.

[0220] Clause 48: The processor of clause 46, wherein the sign and offset corrector component is further configured to add an offset to the stage output so as to modify the stage output from at least one of the at least two successive linear approximators.

[0221] Clause 49: A processor as in any of clauses 41-48, wherein each of the at least two successive linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of the function approximation so as to approximate the linear function using one or more function parameters in the parameter set for each of the at least two successive linear approximators.

[0222] Item 50: A method for processing data using a configurable nonlinear activation function circuit, comprising: determining a parameter set for a selected nonlinear activation function based on the selected nonlinear activation function; and generating output data based on the application of the parameter set for the selected nonlinear activation function, wherein: the configurable nonlinear activation function circuit includes at least one nonlinear approximator, the at least one nonlinear approximator includes at least two consecutive linear approximators, and each linear approximator of the at least two consecutive linear approximators is configured to approximate a linear function using one or more function parameters in the parameter set.

[0223] Clause 51: The method of clause 50, wherein each linear approximator of the at least two successive linear approximators comprises: a stage input; a coefficient input; a constant input; and a stage output.

[0224] Clause 52: The method of any of Clauses 50 or 51, wherein the at least one non-linear approximator comprises a cubic approximator having three successive linear approximators.

[0225] Clause 53: The method of any of Clauses 50-52, wherein the at least one nonlinear approximator comprises a quadratic approximator having two successive linear approximators.

[0226] Clause 54: A method as in clause 51, wherein the at least one nonlinear approximator further comprises a region finder component configured to determine an input value region, and each of the at least two consecutive linear approximators is further configured to determine a coefficient input and a constant input based on the input value region.

[0227] Clause 55: The method of any of clauses 50-54, further comprising: modifying a stage output from at least one of the at least two successive linear approximators using a sign and offset corrector component.

[0228] Clause 56: The method of clause 55, further comprising: inverting the sign of the stage output using a sign and offset corrector component to modify the stage output from at least one of the at least two successive linear approximators.

[0229] Clause 57: The method of clause 55, further comprising: using a sign and offset corrector component to add an offset to the stage output to modify the stage output from at least one of the at least two successive linear approximators.

[0230] Clause 58: A method as in any of clauses 50-57, wherein each of the at least two successive linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of the function approximation so as to approximate the linear function using one or more function parameters in the parameter set for each of the at least two successive linear approximators.

[0231] Clause 59: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any of clauses 18-36 or 50-58.

[0232] Clause 60: A processing system comprising means for performing the method according to any of clauses 18-36 or 50-58.

[0233] Clause 61: A non-transitory computer readable medium comprising computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any of clauses 18-36 or 50-58.

[0234] Clause 62: A computer program product embodied on a computer-readable storage medium, comprising code for performing the method according to any of clauses 18-36 or 50-58.

[0235] Additional considerations

[0236] The previous description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not limitations on the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the universal principles defined herein may be applied to other embodiments. For example, the functions and arrangements of the elements discussed may be changed without departing from the scope of the present disclosure. Various examples may appropriately omit, replace, or add various procedures or components. For example, the described method may be performed in an order different from that described, and various steps may be added, omitted, or combined. Moreover, the features described with reference to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice a method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality that are supplementary to or different from the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the present disclosure disclosed herein may be implemented by one or more elements of the claims.

[0237] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0238] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0239] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Also, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0240] As used herein, in the context of sharing electronic signals and data between the various elements described in this article, the term "connected to" generally refers to the data communication between the corresponding elements connected to each other. In some cases, the elements can be directly connected to each other, such as via one or more conductive traces, lines or other conductive carriers that can carry signals and / or data between the corresponding elements directly connected to each other. In other cases, the elements can be indirectly connected to each other, such as via one or more data buses or similar shared circuit systems and / or integrated circuit elements for conveying signals and data between the corresponding elements indirectly connected to each other.

[0241] Each method disclosed herein includes one or more steps or actions for implementing the method. These method steps and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be changed without departing from the scope of the claims. In addition, the various operations of the above-mentioned methods can be performed by any suitable device capable of performing the corresponding functions. These devices may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the accompanying drawings, these operations may have corresponding paired devices plus functional components with similar numbers.

[0242] The following claims are not intended to be limited to the embodiments shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean "there is and only one" (unless specifically stated), but "one or more". Unless otherwise specifically stated, the term "some / some" refers to one or more. No element of the claim should be interpreted under the provisions of 35 U.S.C. § 112 (f), unless the element is explicitly described using the phrase "device for..." or in the case of a method claim, the element is described using the phrase "step for...". The elements of the various aspects described throughout this disclosure are all structural and functional equivalents currently or hereafter known to those of ordinary skill in the art, and are expressly incorporated herein by reference, and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be donated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A processor, comprising: A configurable nonlinear activation function circuit, wherein the configurable nonlinear activation function circuit is configured to: determining a set of parameters for a selected non-linear activation function based on the selected non-linear activation function; as well as generating output data based on application of said set of parameters for a selected non-linear activation function, in: The configurable nonlinear activation function circuit includes at least one nonlinear approximator, the at least one nonlinear approximator includes at least two consecutive linear approximators, and Each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the parameter set.

2. The processor of claim 1 , wherein each of the at least two successive linear approximators comprises: Level input; Coefficient input; Constant input; as well as Level output.

3. The processor of claim 1, wherein the at least one nonlinear approximator comprises a cubic approximator having three successive linear approximators.

4. The processor of claim 1, wherein the at least one nonlinear approximator comprises a quadratic approximator having two successive linear approximators.

5. The processor of claim 2, wherein: The at least one nonlinear approximator further comprises a region finder component configured to determine a region of input values, and Each linear approximator of the at least two successive linear approximators is further configured to determine the coefficient input and the constant input based on the input value range.

6. The processor of claim 1, wherein the at least one nonlinear approximator further comprises a sign and offset corrector component configured to modify a stage output from at least one of the at least two consecutive linear approximators.

7. The processor of claim 6, wherein the sign and offset corrector component is further configured to invert the sign of the stage output so as to modify the stage output from the at least one of the at least two consecutive linear approximators.

8. The processor of claim 6, wherein the sign and offset corrector component is further configured to add an offset to the stage output so as to modify the stage output from the at least one of the at least two consecutive linear approximators.

9. A processor as claimed in claim 1, wherein each of the at least two consecutive linear approximators is further configured to select the one or more function parameters based at least in part on selecting one or more non-uniform segments of function approximation so as to approximate a linear function using one or more function parameters in the parameter set for each of the at least two consecutive linear approximators.

10. A method for processing data using a configurable nonlinear activation function circuit, comprising: determining a set of parameters for a selected non-linear activation function based on the selected non-linear activation function; as well as generating output data based on application of said set of parameters for a selected non-linear activation function, in: The configurable nonlinear activation function circuit includes at least one nonlinear approximator, the at least one nonlinear approximator includes at least two consecutive linear approximators, and Each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the parameter set.

11. The method of claim 10, wherein each of the at least two successive linear approximators comprises: Level input; Coefficient input; Constant input; as well as Level output.

12. The method of claim 10, wherein the at least one nonlinear approximator comprises a cubic approximator having three successive linear approximators.

13. The method of claim 10, wherein the at least one nonlinear approximator comprises a quadratic approximator having two sequential linear approximators.

14. The method of claim 11, wherein: The at least one nonlinear approximator further comprises a region finder component configured to determine a region of input values, and Each linear approximator of the at least two successive linear approximators is further configured to determine the coefficient input and the constant input based on the input value range.

15. The method of claim 10, further comprising: A stage output from at least one of the at least two successive linear approximators is modified using a sign and offset corrector component.

16. The method of claim 15, further comprising: The sign of the stage output is inverted using the sign and offset corrector component to modify the stage output from the at least one of the at least two consecutive linear approximators.

17. The method of claim 15, further comprising: An offset is added to the stage output using the sign and offset corrector component to modify the stage output from the at least one of the at least two consecutive linear approximators.

18. A method as claimed in claim 10, wherein each of the at least two consecutive linear approximators is further configured to select the one or more function parameters based at least in part on selecting one or more non-uniform segments of function approximation so as to approximate the linear function using one or more function parameters in the parameter set for each of the at least two consecutive linear approximators.

19. A non-transitory computer-readable medium comprising computer-executable instructions, which when executed by a processor comprising a configurable nonlinear activation function circuit causes the processor to perform a method comprising: determining a set of parameters for a selected non-linear activation function based on the selected non-linear activation function; as well as generating output data based on application of said set of parameters for a selected non-linear activation function, in: The configurable nonlinear activation function circuit includes at least one nonlinear approximator, the at least one nonlinear approximator includes at least two consecutive linear approximators, and Each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the parameter set.

20. The non-transitory computer readable medium of claim 19, wherein each of the at least two successive linear approximators comprises: Level input; Coefficient input; Constant input; as well as Level output.

21. The non-transitory computer readable medium of claim 19, wherein at least one nonlinear approximator comprises a cubic approximator having three successive linear approximators.

22. The non-transitory computer readable medium of claim 19, wherein the at least one nonlinear approximator comprises a quadratic approximator having two sequential linear approximators.

23. The non-transitory computer readable medium of claim 20, wherein: The at least one nonlinear approximator further comprises a region finder component configured to determine a region of input values, and Each linear approximator of the at least two successive linear approximators is further configured to determine the coefficient input and the constant input based on the input value range.

24. The non-transitory computer readable medium of claim 19, wherein the method further comprises: A stage output from at least one of the at least two successive linear approximators is modified using a sign and offset corrector component.

25. The non-transitory computer readable medium of claim 24, wherein the method further comprises: The sign of the stage output is inverted using the sign and offset corrector component to modify the stage output from the at least one of the at least two consecutive linear approximators.

26. The non-transitory computer readable medium of claim 24, wherein the method further comprises: An offset is added to the stage output using the sign and offset corrector component to modify the stage output from the at least one of the at least two consecutive linear approximators.

27. A non-transitory computer-readable medium as described in claim 19, wherein each of the at least two successive linear approximators is further configured to select the one or more function parameters based at least in part on selecting one or more non-uniform segments of function approximation so as to approximate a linear function using one or more function parameters in the parameter set for each of the at least two successive linear approximators.

Images