Configurable Nonlinear Activation Function Circuit

Abstract

A processor comprising a configurable non-linear activation function circuit configured to determine a set of parameters for a non-linear activation function based on a selected non-linear activation function and generate output data based on the application of the set of parameters for the non-linear activation function, the configurable non-linear activation function circuit comprising at least one non-linear approximator including at least two sequential linear approximators, each linear approximator of the at least two sequential linear approximators being configured to approximate a linear function using one or more function parameters of the set of parameters.

Description

Technical Field

[0001] (Cross - Reference to Related Applications)

[0001] This application claims the benefit of priority of U.S. Patent Application No. 17 / 807,125, filed Jun. 15, 2022, which is a continuation - in - part of U.S. Patent Application No. 17 / 467,079, filed Sep. 3, 2021, the entire content of which is incorporated herein by reference.

[0002] Introduction

[0002] Aspects of the present disclosure relate to processing a nonlinear activation function for a machine learning model, and more particularly, to a configurable nonlinear activation function circuit.

[0003]

[0003] Machine learning is generally a process of generating a trained model (e.g., an artificial neural network), which represents a generalized fit to a set of training data. Applying the trained model to new data makes it possible to make inferences, which can be used to gain insights into the new data.

[0004]

[0004] As the use of machine learning has increased rapidly to enable various machine learning (or artificial intelligence) tasks, there has been a need for more efficient processing of machine learning model data. In some cases, dedicated hardware such as a machine learning (or artificial intelligence) accelerator or processor or similar circuitry can also be used to enhance the capabilities of a processing system for processing machine learning model data. For example, the processing of data using a nonlinear activation function may be distributed to a processor other than a primary matrix multiplication processor. However, dispersing the various aspects of processing a machine learning model across different processing devices can incur latency, memory usage, power usage, and other processing penalties.

[0005]

[0005] Therefore, an improved technique for processing machine learning model data using a non-linear activation function is needed.

Summary of the Invention

[0006]

[0006] Some aspects provide a processor comprising a configurable non-linear activation function circuit configured to determine a non-linear activation function for applying to input data, determine a set of parameters for the non-linear activation function based on the determined non-linear activation function, and generate output data based on application of the set of parameters for the non-linear activation function.

[0007]

[0007] A further aspect provides a method for processing input data by a configurable non-linear activation function circuit, the method comprising determining a non-linear activation function for applying to the input data, determining a set of parameters for the configurable non-linear activation function circuit based on the determined non-linear activation function, and processing the input data using the configurable non-linear activation function circuit based on the set of parameters to generate output data.

[0008]

[0008] A further aspect provides a processor comprising a configurable non-linear activation function circuit configured to determine a set of parameters for a non-linear activation function based on a selected non-linear activation function, and to generate output data based on the application of the set of parameters for the non-linear activation function, wherein the configurable non-linear activation function circuit comprises at least one non-linear approximator comprising at least two successive linear approximators, and each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the set of parameters.

[0009]

[0009] A further aspect provides a method for processing data using a configurable non-linear activation function circuit, the method comprising determining a set of parameters for a non-linear activation function based on a selected non-linear activation function, and generating output data based on the application of the set of parameters for the non-linear activation function, wherein the configurable non-linear activation function circuit comprises at least one non-linear approximator comprising at least two successive linear approximators, and each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the set of parameters.

[0010]

[0010] Another aspect provides a processing system configured to implement the methods described above and the methods described herein, a non-transitory, computer-readable media including instructions that, when executed by one or more processors of the processing system, cause the processing system to implement the methods described above and the methods described herein, a computer program product embodied on a computer readable storage medium comprising code to implement the methods described above and the methods further described herein, and a processing system comprising means for implementing the methods described above and the methods further described herein.

[0011]

[0011] The following description and the related drawings detail some exemplary features of one or more embodiments.

Brief Description of the Drawings

[0012]

[0012] The accompanying drawings illustrate some aspects of one or more embodiments and, accordingly, should not be considered as limiting the scope of the disclosure.

Figure 1

[0013] An exemplary configurable nonlinear activation (CNLA) function circuit is shown.

Figure 2

[0014] An exemplary circuit block for implementing a bypassable approximator block as described with respect to FIG. 1 is shown.

Figure 3

[0015] An exemplary approximator is shown.

Figure 4

[0016] An exemplary process flow of a machine learning model is shown.

Figure 5

[0017] An exemplary method for performing processing using a configurable nonlinear activation function circuit is shown.

Figure 6

[0018] An example of a pipelined sequential linear approximation architecture that can be used to implement a non - linear function used in the acceleration of a machine learning model is shown.

Figure 7

[0019] An example of a hyperbolic tangent (tanh) non - linear activation function with a non - uniform segment for defining an approximated function output is shown.

Figure 8

[0020] An example of a circuit for implementing a pipelined sequential linear approximation architecture is shown.

Figure 9

[0021] An example of a linear approximator circuit that can be an example of a linear approximator element within the circuit of FIG. 8 is shown.

Figure 10

[0022] An exemplary processing system configured to execute the methods described herein is shown.

[0013]

[0023] For ease of understanding, where possible, the same reference numbers are used to designate the same elements common to the drawings. Advantageously, it is contemplated that the elements and features of one embodiment can be incorporated into other embodiments without further recitation.

DETAILED DESCRIPTION OF THE INVENTION

[0014]

[0024] Aspects of the present disclosure provide improved techniques for processing non - linear activation functions associated with machine learning models.

[0015]

[0025] Nonlinear activation is an important component of various types of machine learning models, including neural network models. Some nonlinear activation functions are implemented as piecewise linear functions (e.g., ReLU (rectified linear unit), leaky ReLU, etc.), while other nonlinear activation functions require complex mathematical functions (e.g., sigmoid, hyperbolic tangent (tanh), etc.). In some cases, the complex mathematical functions may be implemented using interpolation such as cubic spline interpolation. For example, the interpolated output value can be determined using a look-up table (LUT) in some manners to match the output value with the input value. When the target input value is not mapped in the LUT, the LUT values associated with the input values adjacent to the target input value can be used to interpolate the output value for the target input value.

[0016]

[0026] Conventionally, due to the wide range of possible activation functions available for use in machine learning models, nonlinear activation functions could be implemented in software rather than hardware. However, such an implementation typically requires moving model data between processing devices (e.g., between a neural processing unit (NPU) that performs matrix multiplication and accumulation and a digital signal processor (DSP) that processes the nonlinear activation function), thus incurring power and latency penalties. When nonlinear activation functions are implemented in hardware, they are generally limited to supporting only a small number of nonlinear activation functions, and thus cannot be configured to support evolving machine learning model architectures without delegating the processing of the nonlinear activation functions to distributed processing units.

[0017]

[0027] For example, the rectified linear unit (ReLU) is an activation function commonly used in deep learning models. This function returns 0 when receiving a negative input, and otherwise returns the input x. Therefore, it can be written as f(x) = max(0, x). The ReLU function is generally not implemented by units for matrix multiplication and accumulation operations such as compute-in-memory (CIM) arrays in some examples. Therefore, the need to distribute the ReLU function or other non-linear activation functions is costly from a processing perspective. Furthermore, as the activation function becomes more complex, the processing cost (for example, for performing relatively higher power exponents and division operations that are part of some non-linear activation functions, as will be further described below) also becomes similarly more important.

[0018]

[0028] To overcome the drawbacks of conventional solutions, the aspects described herein relate to a configurable non-linear activation (CNLA) function circuit that can be implemented in hardware for efficient processing. In particular, since the CNLA function circuit can be implemented in hardware, the CNLA function can be placed in the same location as other processing circuits optimized for other machine learning model processing tasks, such as CIM arrays and digital multiply-and-accumulate (DMAC) circuits optimized to perform vector and matrix multiplication and accumulation functions.

[0019]

[0029] To improve processing efficiency, the aspects described herein may use polynomial approximation to approximate complex functions, such as those that can be used within a non-linear activation function. In some cases, the aspects described herein can use a series expansion, such as a Taylor series. Generally, the Taylor series of a function (e.g., f(x)) is an infinite sum of terms expressed in terms of the derivatives of the function at a single point. For many functions, the sum of the function and its Taylor series is equal near this point. The partial sum formed by the first n + 1 terms of the Taylor series is a polynomial of degree n, called the n-th Taylor polynomial of the function. Thus, the Taylor polynomial enables efficient approximation of the function, which generally improves as n increases.

[0020]

[0030] The CNLA function circuit described herein can generally implement one or more polynomial approximation blocks, such as a cubic approximation block, to improve cubic spline interpolation to be more efficient, more generalized, and cover a greater variety of non-linear activation functions. Additionally, the CNLA function circuit may be implemented as a pipelined digital block that can use non-linearly segmented look-up tables (LUTs) and mixed-degree approximations (e.g., pipelined linear approximation, quadratic approximation, and cubic approximation). Thus, the CNLA function circuit described herein can be configured to meet many different performance goals, unlike conventional non-linear activation function circuits.

[0021]

[0031] Thus, the CNLA function circuit described herein provides a technical solution to the technical problem of implementing a wide range of non-linear activation functions in a machine learning model processing system. Additionally, the CNLA function circuit described herein provides technical improvements, including lower latency, lower power usage, improved memory efficiency, and other advantages described herein, by improving model processing performance compared to existing solutions.

[0022] Exemplary Configurable Non-Linear Activation Function Circuit

[0032] FIG. 1 shows an exemplary configurable non - linear activation (CNLA) function circuit 100.

[0023]

[0033] Generally, the CNLA function circuit 100 can be configured to receive input data 101 (e.g., an output value from a layer of a machine learning model) and execute various non - linear activation functions to generate output data 114 (e.g., “activation”). The CNLA function circuit 100 can be placed in the same location as other machine learning model processing circuits such as a CIM array, DMAC, etc., and may be pipelined and configured to execute an activation function based on the output of other machine learning model processing circuits.

[0024]

[0034] In some examples, the input data 101 can be received from a buffer or other memory. In other examples, the input data 101 can be received directly from the output of another processing block such as the output of a CIM array or another vector and matrix multiplication and accumulation block.

[0025]

[0035] The CNLA function circuit 100 includes a first approximator block 102 that can generally be configured to execute a hardware - based mathematical function on the input data 101, etc. An exemplary approximator is described in detail with respect to FIG. 3.

[0026]

[0036] In some cases, the first approximator is a linear approximator (configured to execute a function such as ax + b), a quadratic approximator (configured to execute a function such as ax 2 + bx + c), (e.g., a function configured to execute a function such as ax 3 + bx 2One of the cubic approximators configured to execute a function such as ax³ + bx² + cx + d, where x is the input data and a, b, c, and d are configurable parameters. In general, a linear approximator, quadratic approximator, or cubic approximator can be used to approximate some given function (which may or may not be a polynomial function). The first approximator 102 can be set, for example, by parameters retrieved from memory, registers, look-up tables, etc. As will be described in more detail below with respect to Table 2, many types of non-linear activation functions can be approximated using these different forms of approximation and the associated configurable parameters.

[0027]

[0037] The CNLA function circuit 100 further includes a second approximator block 104 that can generally be configured to execute a hardware-based mathematical function such as a linear function, quadratic function, or cubic function, similar to the first approximator block 102. As will be described in more detail below, the CNLA function circuit 100 can be configured to use the first approximator block 102 and the second approximator block 104 in series for more complex functions such that the output of the first approximator block 102 becomes the input to the second approximator block 104. The CNLA function circuit 100 can be further configured to use only one of the first approximator block 102 or the second approximator block 104 when a simpler non-linear function is being processed, thereby saving power.

[0028]

[0038] In some implementations, the first approximator 102 and the second approximator 104 may comprise the same circuit block (e.g., two instances of the same circuit elements within circuit 100). For example, the first approximator 102 and the second approximator 104 may comprise a third-order approximator in some aspects. In other implementations, the first approximator 102 and the second approximator 104 may comprise different circuit elements, and in such cases, generally, the second approximator 104 comprises a third-order approximator and the first approximator 102 comprises a lower-order approximator such as a second-order or linear approximator. However, in other embodiments, the order of the higher-order approximator and the lower-order approximator can be reversed.

[0029]

[0039] The CNLA function circuit 100 includes a configurable bypass 105 that enables bypassing the first approximator 102 in various scenarios, such as when the function requires only an approximator of a lower degree than the first approximator 102 and the second approximator 104 is such a lower-degree approximator. For example, when the first approximator 102 is bypassed via the configurable bypass 105, the input data 101 is instead provided directly to the second approximator 104 and is not processed by the first approximator 102. In various aspects, the first approximator 102 can be a higher-order approximator compared to the second approximator 104, or vice versa, or they can be of the same degree (e.g., both linear, second-order, or third-order). The configurable bypass 105 enables saving processing time and energy when only one approximator is needed.

[0030]

[0040] The CNLA function circuit 100 further includes another configurable bypass 107 that enables bypassing the second approximator 104 in various scenarios, such as when the function requires only the first approximation, and the first approximator 102 can execute without the second approximator 104. For example, when the second approximator 104 is bypassed via the configurable bypass 107, the output of the first approximator 102 is provided directly to the multiplier 108.

[0031]

[0041] Generally, the configurable bypasses 105 and 107 enable the CNLA function circuit 100 to be configured to have maximum versatility, while saving power and avoiding unnecessary circuit block processing in various scenarios. Further, the configurable bypasses enable the non - symmetric and anti - symmetric non - linear activation functions to be set to be processed by the CNLA function circuit 100. FIG. 2 shows an exemplary circuit aspect for implementing the configurable bypasses 105 and 107 (e.g., bypasses 205A and 205B).

[0032]

[0042] The CNLA function circuit 100 further includes a gain block 106 configured to provide a gain value to a multiplier 108. In some aspects, the gain block 106 is configured to generate a gain value 109 based on a gain function implemented by the gain block 106. In one example, the gain function may be in the form of g = ax + b, where g is the gain value, x is the value of the input data 101, and a and b are configurable parameters. More generally, the gain block 106 can modify the input data multiplicatively (a) and / or additively (b) to generate a gain value.

[0033]

[0043] The gain value 109 generated by the gain block 106 is multiplied via the multiplier 108 with the output of the first approximator 102 and / or the second approximator 104. In other aspects, the gain block 106 can be configured to generate a gain value not based on a function of the input data 101 (e.g., by setting a to 0 in the above - mentioned equation for g). Generally, the parameters (e.g., a and b in the above example) or values of the gain block 106 can be retrieved from, for example, memory, registers, look - up tables, etc.

[0034]

[0044] The CNLA function circuit 100 further includes a constant block 110 configured to store a configurable (e.g., programmable) constant value 113, and an adder 112 configured to add the constant value 113 to the output of a multiplier 108 (e.g., a gain multiplier). The constant value 113 stored in the constant block 110 can be retrieved from, for example, a memory, a register, a look-up table, etc.

[0035]

[0045] The inclusion and arrangement of the first approximator block 102, the second approximator block 104, the configurable bypasses 105 and 107, the gain block 106, the multiplier 108, the constant block 110, and the adder 112 enable the CNLA function circuit 100 to be configured to execute a wide variety of known and future-developed non-linear activation functions. Further, the CNLA function circuit 100 can be efficiently configured to process a wide variety of non-linear activation functions simply by updating the parameters of the first approximator 102, the second approximator 104, the gain block 106, and the constant block 110. When both the approximator blocks 102 and 104 are used to simulate non-linear functions, each approximator block can be said to execute an individual function (e.g., a first function for the first approximator block 102 and a second function for the second approximator block 104). This design advantageously supports any asymmetric non-linear curve for complex functions.

[0036]

[0046] Table 1 below provides exemplary parameters for various non-linear activation functions that the CNLA function circuit 100 of FIG. 1 can be configured to execute, including the parameters for the approximator blocks 206A and 206B of FIG. 2. In Table 1, the gain is considered to have the form ax + b, similar to the example of the gain block 106 in FIG. 1, but note that in other embodiments, the gain can be a scalar value or a different functional form. Similarly, the quadratic approximator is considered to have the form ax 2 + bx + c, and the cubic approximator is ax 3 + bx 2It is considered to have the form of +cx + d. In the following table, subscripts are used to indicate parameter assignments. For example, G is used for the gain parameter, 1 is used for the first approximator, and 2 is used for the second approximator parameter.

[0037]

Table 1-1

[0038]

Table 1-2

[0039]

[0047] It should be noted that in the above ELU function, the α parameter may be set as a hyperparameter by the model designer.

[0040]

[0048] In particular, in some implementations, even when the approximator is performing a lower-order function (e.g., linear), the parameters of the approximator can be given in a form (e.g., cubic with a, b, c, and d parameters, or quadratic with a, b, and c parameters). This is because, for example, setting the cubic parameter a to 0 causes the approximation formula to effectively collapse to a lower-order quadratic function, and similarly, setting the quadratic parameter a to 0 causes the approximation formula to effectively collapse to a linear equation. Thus, the approximator can be set for a "quadratic function" when using, for example, quadratic parameters, but the result of the parameters can reduce the function to a linear function, as in the ReLU example in Table 2 above. This enables the normalization of the parameter set regardless of the degree of the underlying function set by the parameters, thereby simplifying the implementation.

[0041]

[0049] FIG. 2 shows exemplary circuit blocks 202 and 204 for implementing bypassable approximator blocks 206A and 206B. The bypassable approximator blocks 206A and 206B may correspond, in one example, to the first approximator block 102 and the second approximator block 104 of FIG. 1.

[0042]

[0050] In FIG. 2, circuit block 202 is configured to control the use of functional block 214A, which in this example includes first approximator 206A and minimum and maximum function blocks 208A. Similarly, circuit block 204 controls the use of functional block 214B, which in this example includes minimum and maximum function blocks 208B and second approximator 206B. The first and second approximator blocks 206A and 206B can be set to implement non-linear activation functions, such as those described above with respect to Table 1.

[0043]

[0051] Note that while the first approximator 102 of FIG. 1 requires only one input, circuit block 202 includes two input ports 201A and 201B, which allows for multiple inputs. The illustrated configuration of circuit block 202 can be employed to present the same external interface for both circuit blocks 202 and 204, which can simplify configuration and integration. In some embodiments, the two input ports 201A and 201B of circuit block 202 can be coupled to each other in an implementation where circuit block 202 receives a single input (such as input data 101 of FIG. 1) via input port 201A. In an alternative implementation, circuit block 202 can be simplified by removing input port 201B and input MUX 203A such that 201A is provided directly to 214A and 207A.

[0044]

[0052] In general, input ports 201A and 201B can receive various types of input data for processing, including signed multi-bit integer data. In one example, the input data is 8-bit two's complement input data.

[0045]

[0053] The input selectors MUX203A and MUX203B are configured to control which input data ports are used for circuit blocks 202 and 204 respectively. For example, the input selector MUX203B can select between input data port 201A (e.g., when circuit block 202 is bypassed) or 212B (e.g., when circuit blocks 202 and 204 are processed in series).

[0046]

[0054] The bypass selectors MUX211A and MUX211B are configured to control the bypass function blocks 214A and 214B of circuit blocks 202 and 204 respectively. For example, when circuit block 202 is bypassed, the bypass selector MUX211A selects the bypass line 205A to provide an output to output port 212A. Similarly, when circuit block 204 is bypassed, the bypass selector MUX211B selects the bypass line 205B to provide an output to output port 216. In this way, the processing using circuit blocks 202 and / or 204 controlled by the configurable bypasses 205A and 205B results in an output at output port 216.

[0047]

[0055] As will be described in more detail with respect to FIG. 3, the approximator blocks 206A and 206B can be set respectively using set parameters (e.g., function specific coefficients as in Table 1 above) stored in registers 219A and 219B. Similarly, as in Table 1 above, when the approximator block 206A or 206B is configured to perform a look-up table based function, the table values can be stored in registers 219A and 219B respectively.

[0048]

[0056] Each circuit block (202 and 204) further includes a minimum / maximum function block (208A for circuit block 202 and 208B for circuit block 204) for providing minimum and maximum functions. Generally, a minimum (or "min") function returns the minimum value among the provided inputs. Similarly, a maximum (or "max") function returns the maximum value among the provided inputs. In one example, the minimum / maximum function blocks 208A and 208B may comprise a multi-bit digital comparator operating in either single-cycle mode or multi-cycle mode.

[0049]

[0057] The settings of the function blocks 214A and 214B may each include settings for the function selector MUXes 209A and 209B. In other words, whether the function blocks 214A and 214B output the min / max outputs from the mix / max blocks 208A and 208B or the values from the approximators 206A and 206B is based on the settings of the function selector MUXes 209A and 209B. Note that in other examples, the function blocks 214A and 214B may include additional function blocks that can be selected by a MUX.

[0050]

[0058] As shown in FIG. 1, when the approximator blocks can be processed in series, in FIG. 2, the output 212A of the circuit block 202A including the first approximator block 206A is provided as an input 212B to the circuit block 204 including the second approximator block 206B. Similar to FIG. 1 where bypasses 105 and 107 control the use of the first and second approximator blocks 102 and 104, here, selectable bypasses 205A and 205B control the use of the approximator blocks 206A and 206B.

[0051]

[0059] The asymmetric signal line 210 controls the settings of circuit blocks 202 and 204. In one example, circuit blocks 202 and 204 are set based on the values on the asymmetric signal line 210 and the output values from code blocks 207A and 207B based on the input data received via input data port 201A. For example, the binary value received via the asymmetric signal line 210 and the binary value output from code block 207A interact in AND gate 213 to control the selection of the output by MUX211A. As another example, the binary value received via the asymmetric signal line 210 and the binary value output from code block 207B interact in AND gate 217 to control the selection of the input data port (such as between 201A and 212B) via MUX203B. As a further example, the binary value received via the asymmetric signal line 210 and the inverted binary value output from code block 207B interact in gate 215 to control the selection of output MUX211B.

[0052]

[0060] Table 2 below provides an overview of the settings of circuit blocks 202 and 204.

[0053]

Table 2

[0054] Exemplary approximator for a configurable non - linear activation function circuit

[0061] FIG. 3 shows an exemplary approximator 300, which can be an example of one or both of the first approximator 102 and the second approximator 104 of FIG. 1, and / or the approximators 206A and 206B of FIG. 2.

[0055]

[0062] The approximator 300 receives input data 302 (e.g., pre-activation data) for processing. In some examples, the input data 302 can be received from a buffer or other memory. In other examples, the input data can be received directly from the output of another processing block, such as a CIM array or the output of another vector and matrix multiplication and accumulation block. Additionally, the input data can be received from another approximator, such as when the approximator 300 is the second approximator 104 of FIG. 1 or the second approximator 206B of FIG. 2.

[0056]

[0063] In some implementations, an approximator (such as 300) can include alternative processing paths. In such cases, the path logic 304 can be configured to route the input data 302 to an appropriate processing path, for example, based on the configuration parameters for the approximator 300.

[0057]

[0064] In this example, the processing path 306A provides a cubic approximation path for the input data 302.

[0058]

[0065] In the processing path 306A, the input data 302 is supplied to a cubic calculator 308, which performs a cubic operation (e.g., x 3 , where x is the input data), and then the output is multiplied by a cubic parameter 312 in a multiplier 310. The output of the multiplier 310 is then supplied to an accumulator 324.

[0059]

[0066] The input data 302 is also supplied to a quadratic calculator 308, which performs a quadratic operation (e.g., x 2 , where x is the input data), and then the output is multiplied by a quadratic parameter 318 in a multiplier 316. The output of the multiplier 316 is then supplied to the accumulator 324.

[0060]

[0067] The input data 302 is also supplied to a multiplier 320, where it is multiplied by a linear parameter 322. The output of the multiplier 320 is then supplied to the accumulator 324.

[0061]

[0068] The accumulator (adder) 324 accumulates the outputs of the multipliers 310, 316, and 320, as well as the intercept parameter 326, to generate the output data 332.

[0062]

[0069] The cubic parameter 312, the quadratic parameter 318, the linear parameter 322, and the intercept parameter 326 can all be stored (e.g., in a register) in a memory accessible from the accumulator 300. In some cases, a control unit such as a memory control unit or a finite state machine can set the approximator 300 using the parameters stored in the memory. In various examples, the cubic parameter 312, the quadratic parameter 318, the linear parameter 322, and the intercept parameter 326 can be set according to the values described above with respect to Table 2.

[0063]

[0070] Thus, by setting the above-described parameter values, the degree of approximation can be set. For example, the cubic parameter 312 can be set to 0 for the approximator 300 to perform a quadratic approximation. Similarly, the cubic parameter 312 and the quadratic parameter 318 can be set to 0 for the approximator 300 to perform a linear approximation.

[0064]

[0071] Certain non-linear activation functions require alternative functions such as a minimum function and a maximum function. Thus, the processing path 306B provides, for example, a minimum calculator and / or a maximum calculator that can be used with the ReLU and ReLU6 functions described above in Table 2. The processing path 306B may be selected by the path logic 304 based on the configuration data of the approximator 300.

[0065]

[0072] Furthermore, certain non-linear activation functions may be implemented using a look-up table, which provides a more power- and time-efficient mechanism for generating the values of the certain non-linear activation function. Thus, processing path 306C provides a look-up table-based processing path that may be used, for example, when a sigmoid function, a tanh function, or a similar function is used by the non-linear activation function. Note that sigmoid and tanh can be calculated from each other, and thus in some cases only a single look-up table (e.g., sigmoid or tanh, but not both) is stored and used to implement both functions. One or more look-up tables may be stored in memory and be accessible from approximator 300, which includes memory tightly coupled to approximator 300.

[0066] Process flow of an exemplary machine learning model using a configurable non-linear activation function circuit

[0073] FIG. 4 shows a data flow 400 of an exemplary machine learning model implementing a configurable non-linear activation function circuit as described above with respect to FIGS. 1-3.

[0067]

[0074] In flow 400, the input data is stored in the input data buffer 401 (e.g., the layer input data of the machine learning model) and then supplied to the multiply and accumulate (MAC) circuit 402. The MAC circuit 402 can generally be configured to perform vector, array, and matrix multiply and accumulate operations, such as those frequently used in convolutional neural networks. In some examples, the MAC circuit 402 may include one or more compute-in-memory (CIM) arrays. Alternatively, or in addition, the MAC circuit 402 may include digital multiply and accumulate (DMAC). In further examples, the multiply and accumulate circuit 402 can be part of a machine learning accelerator such as a neural processing unit (NPU), or another type of processing unit optimized for performing machine learning processing. In another implementation, the MAC circuit 402 can be replaced by a vector / matrix or matrix / matrix processing engine.

[0068]

[0075] The MAC circuit 402 processes the input data using weight data (e.g., neural network weight data) to generate pre-activation data. For example, the MAC circuit 402 can process the input data to a layer of the neural network model and generate pre-activation data as the output.

[0069]

[0076] The pre-activation data is supplied to a configurable non-linear activation (CNLA) function circuit 404 configured to generate output data (e.g., activation) based on a set non-linear activation function. The output data can then be stored in the output data buffer 405 for subsequent use, such as for processing another layer within the machine learning model or as the output from the machine learning model.

[0070]

[0077] The CNLA function circuit 404 may be set using the setting parameters as described for the CNLA function circuit 300 in FIG. 3, as well as the setting parameters described in Tables 1 and 2. Further, the CNLA function circuit 404 may be set to access a look-up table according to the set activation function.

[0071]

[0078] In some cases, the setting parameters may include the identification of a non-linear activation function applied to the input data. Based on the determined non-linear activation function, appropriate parameters (such as the parameters in Table 2) are retrieved from a memory (e.g., a register) and applied to the CNLA function circuit 404, thereby enabling the CNLA function circuit to be set to process the input data. In some examples, a finite state machine, a memory control unit, or another controller may perform the setting of the CNLA function circuit 404.

[0072]

[0079] In particular, the CNLA circuit 404 may be configured to process multiple batches of input data using the same setting, or its setting may be updated for each new batch of input data. Thus, the CNLA circuit 404 provides a very flexible and efficient means for performing settable non-linear activation for machine learning tasks such as training and inference.

[0073] Exemplary method for performing processing using a settable non-linear activation function circuit

[0080] FIG. 5 shows an exemplary method 500 for performing processing using a settable non-linear activation function circuit.

[0074]

[0081] Method 500 begins in step 502 by selecting a non-linear activation function to apply to the input data. For example, the non-linear activation function may be one of the functions listed in Table 2, or another non-linear activation function.

[0075]

[0082] Next, method 500 proceeds to step 504 to determine a set of parameters for the configurable non-linear activation function circuit based on the selected non-linear activation function. For example, the determined parameters of the non-linear activation function may be the parameters as in Table 1 and Table 2 above.

[0076]

[0083] Next, method 500 proceeds to step 506 to process the input data with the configurable non-linear activation function circuit based on the set of parameters to generate output data. For example, the output data may be the activation data of a layer of the neural network model.

[0077]

[0084] In some examples, the set of parameters includes a combination of one or more gain parameters, constant parameters, and one or more approximation functions for applying to the input data via the configurable non-linear activation function circuit. For example, the set of parameters can be as described above with respect to FIGS. 1 and 2 and Table 1.

[0078]

[0085] In some examples, method 500 further includes retrieving the set of parameters from memory based on the determined non-linear activation function. In some examples, the memory can be one or more registers that store parameter values.

[0079]

[0086] In some examples, the configurable non-linear activation function circuit includes a first approximator configured to approximate a first function of one or more approximation functions, a second approximator configured to approximate a second function of one or more approximation functions, a first gain multiplier configured to multiply a first gain value based on one or more gain parameters, and a constant adder configured to add a constant value as illustrated and described with respect to FIG. 1.

[0080]

[0087] In some examples, a configurable non-linear activation function circuit includes a first bypass configured to bypass a first approximator. In some examples, a configurable non-linear activation function circuit includes a second bypass configured to bypass a second approximator. In some examples, a configurable non-linear activation function circuit includes an input data bypass configured to bypass a first approximator and supply input data to a second approximator.

[0081]

[0088] In some examples, at least one of the first approximator and the second approximator is a cubic approximator. In some examples, the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator. In some examples, the other of the first approximator and the second approximator is configured to execute a min / max function as shown with respect to path 306B in FIG. 3. In some examples, the other of the first approximator and the second approximator is configured to access a look-up table for approximations as shown with respect to path 306C in FIG. 3.

[0082]

[0089] In some examples, both the first approximator and the second approximator are cubic approximators.

[0083]

[0090] Note that FIG. 5 is only an example, and in other examples, methods such as those described herein may be implemented with more steps, fewer steps, and / or different steps.

[0084] Exemplary Pipeline Approximation for Non-Linear Functions

[0091] FIG. 6 shows an example of a pipelined sequential linear approximation architecture 600 that can be used to implement non-linear functions used in the acceleration of a machine learning model. For example, architecture 600 can be used to implement approximators 102 and 104 described with respect to FIG. 1, and approximators 206A and 206B described with respect to FIG. 2.

[0085]

[0092] Generally, FIG. 6 shows an input x flowing in a pipeline manner through linear approximators 604A to 604C. Architecture 600 is a power - efficient and space - efficient way to implement a cubic approximation using sequential linear approximation blocks 604A to 604C. Specifically, the input x is processed by linear approximation block 604A to determine an output y1 = Ax + B, where A and B are coefficients 605A used by linear approximation block 604A.

[0086]

[0093] Next, y1 is provided as an input to linear approximator block 604B, and an output y2 = x(y1)+C is determined. Here, C is coefficient 605B used by linear approximation block 604B. Note that x(y1) represents y1 multiplied by x.

[0087]

[0094] Next, y2 is provided as an input to linear approximator block 604C, and an output y3 = x(y2)+D is determined. Here, D is coefficient 605C used by linear approximation block 604C. By expanding y3 based on sequential linear approximation, the following is obtained. y3=x(y2)+D =x(x(Ax + B)+C)+D =Ax 3 +Bx 2 +Cx + D

[0088]

[0095] This is a cubic approximation based on the input x. In particular, in the sequential linear approximation method implemented in architecture 600, the number of required multipliers is reduced, thereby reducing complexity and power consumption compared to conventional cubic approximator architectures. Furthermore, architecture 600 enables pipeline implementation, and in pipeline implementation, a new output is generated every cycle in the steady state, thus improving throughput.

[0089]

[0096] To further improve processing efficiency, coefficients A, B, C, and D (605A - 605C) can be stored in registers rather than, for example, SRAM or DRAM, thereby improving the power and latency performance of architecture 600. For example, the use of read power from the SRAM of the host system can be avoided by using local registers to store coefficient values. The register approach also supports the pipeline architecture by avoiding additional cycles for reading and transferring coefficient values from the memory (e.g., DRAM) of the remote host system. In various embodiments, the pipeline registers can be internal or external to linear approximators 604A - 604C.

[0090]

[0097] In some implementations, as shown with respect to architecture 600, the coefficients can be determined based on an approximation of a non - linear function that is non - uniformly segmented, as illustrated in an example with respect to FIG. 7. In such embodiments, region finder 602 can determine appropriate coefficients based on regions (or segments) of the non - uniformly segmented approximation according to input x. Thus, FIG. 6 shows the output "REGION_SEL" of the region finder passed in a pipelined fashion through various stages of linear approximation.

[0091]

[0098] Note that the "regions" discovered by region finder 602 are segments of the "domain" of the original non - linear function. That is, a non - linear function can be defined over a domain from negative infinity to positive infinity, and that domain can be segmented. In various aspects described herein, the segments are of non - uniform length. For each such segment, coefficients are selected to best approximate the function within that segment. FIG. 8 shows an exemplary circuit implementation having region finder element 809, where "cut - off" is used to describe the end points of the segments.

[0092]

[0099] In various aspects, the region finder 602 can be configured to select an entire set of coefficients used for the approximate whole. In the illustrated example, since the approximation is performed continuously in a pipeline manner, the determined regions will be executed and registered along the data paths of each approximator, and thus the determined regions can be used for subsequent coefficient lookups at each stage (e.g., in the linear approximators 604A - 604C).

[0093]

[0100] The region finder 602 can generally use various offline algorithms to explore and set region options, and thus does not need to be limited in implementation by how the regions are found. For example, a cubic spline algorithm can be used to ensure a continuous derivative across the region boundaries, which is advantageous for model training. In another example, the mean squared error can be minimized for each point (based on subsequent exploration for the best cubic coefficients for the region under consideration). This second approach can achieve better accuracy in calculations at the expense of a continuous derivative (to achieve better accuracy for inference). Yet another approach is to set regions based on where the second or third derivative is zero, but this approach may not use the complete set of possible regions as more regions can result in less error.

[0094]

[0101] Finally, the architecture 600 is configured to advantageously reduce circuit area and power during processing by leveraging the odd and even symmetry properties of common neural network non-linear functions such as sigmoid and hyperbolic tangent (tanh). Advantageously, by exploiting such symmetry, it becomes possible to reduce the number of coefficients that need to be stored to approximate the non-linear function, because, for example, by a simple sign inversion, both sides of the function from the origin (e.g., positive and negative) can be emulated without the need to store the coefficients on both sides. In some implementations, the approximation results are only calculated for positive inputs, in which case the same result can be used for negative inputs of the same magnitude (even symmetry), or in the case of a function with odd symmetry, the sign of the result can be inverted.

[0095]

[0102] In the illustrated example, the sign and offset corrector component 606 is configured to receive as inputs the third-order approximation y3, the original input x (e.g., to determine the sign), and the offset value 607, and apply sign and / or offset correction to generate the final output 608. For example, the final output 608 may be calculated as offset (607) + / - y3. Similar to the coefficients 605A - 605C, the offset 607 can be stored in a local register for efficiency and speed.

[0096]

[0103] Generally, the offset 607 can be used for functions that are symmetric except that they are not centered around x = 0. One example is the sigmoid, which is symmetric with respect to the line y = 0.5. In such a case, a "shifted" sigmoid that is odd symmetric with respect to the x-axis is first approximated, and then an offset of 0.5 is added to the result.

[0097]

[0104] Note that FIG. 6 shows an exemplary architecture 600 that includes three linear approximators (604A-604C) that can be used to approximate a cubic non-linear function. In other embodiments, two linear approximators can be used if a quadratic non-linear function is intended to be approximated. Although not shown in FIG. 6, in some embodiments, the control logic can switch the output of a linear approximator, such as linear approximator 604B, and bypass a subsequent linear approximator (such as linear approximator 604C) when only a quadratic function needs to be approximated. In such a case, the bypass can take the output y3 of linear approximator 604B and route it to the sign-offset correction component 606.

[0098]

[0105] FIG. 7 shows an example 700 of a hyperbolic tangent (tanh) non-linear activation function having non-uniform segments for defining an approximated function output.

[0099]

[0106] In the illustrated example, the first segment 702A is defined from the origin (x = 0) to a first point along the curve 701. As shown, segments such as 702B are longer along the curve 701 and the gradient is more constant over the input values within the segment, and segments such as 702C are shorter along the curve 701, and this curve changes the gradient more rapidly along the input values within the segment. By using non-uniform segments, it is advantageously possible to define fewer segments over the output value range of the non-linear function while still maintaining a comparable output resolution compared to the actual function. Conventionally, to maintain the resolution, a curve such as 701 is uniformly segmented, which means that many segments have very similar output values and thus redundant values are generated that are stored in memory.

[0100]

[0107] Note that each segment can generally be defined by a starting point and an ending point along the curve 701. For example, the starting input value x start and the ending input value x endcan define segments. Each of these segments can be associated with a set of coefficients for use in approximating the hyperbolic tangent function based on input values falling within a defined range (x start ≦x<x end etc.). To further save memory, the first segment 702A only needs to remember the end point of the segment, because the starting point (origin) can be assumed to be x = 0. Similarly, the last segment is only defined by the end point of the second-to-last segment. That is, all inputs above the end point of the second-to-last segment are considered part of the last segment.

[0101]

[0108] For example, the following table shows an example of non-uniform segments that can be stored in registers for the approximated tanh function. When the input value x falls within a particular segment (or region), the coefficients for that segment can be retrieved and used to generate an approximated function value.

[0102]

Table 3

[0103]

[0109] Note that the hyperbolic tangent function curve 701 is an example of a function with odd symmetry. That is, the tanh output value for a positive input (e.g., 0.2 for point 704A) is equal to the output with the sign reversed for a negative input of the same magnitude (e.g., -0.2 for 704B). Therefore, the non-uniform segments only need to be defined for the positive input value range of the hyperbolic tangent function curve 701, and a sign corrector component such as 606 in FIG. 6 can appropriately reverse the sign of the final output (e.g., 608 in FIG. 6) based on the sign of the input value (x).

[0104]

[0110] FIG. 8 shows an example of a circuit 800 for implementing a pipelined successive linear approximation architecture as described above with respect to FIG. 6. In particular, FIG. 8 shows the pipelined nature of the design, including how the output of the region finder 809 and the sign bits pass through the pipeline for efficient processing.

[0105]

[0111] The illustrated example includes four stages 806A - 806D, three of which include linear approximators (804A - 804C respectively) and one sign / offset correction stage 806D.

[0106]

[0112] In the illustrated example, various operation blocks are shown. "SAT" represents the "saturation" block. Saturation is useful when reducing the number of integer bits used to represent a value. When a saturation module is used to reduce an input having M integer bits to an output having N integer bits (where N < M), input values larger than what can be represented using N integer bits are "saturated" to either the largest positive N - bit number (for positive inputs) or the smallest negative N - bit number (for negative inputs). For example, if [x > 0111...11], then y(0111...11) is output, and if [x <= 1000...00], then y(1000...00) is output.

[0107]

[0113] "ABS" represents a block that calculates the absolute value (or magnitude) of the input.

[0108]

[0114] "COMP" represents a comparator block that acts as a region finder using a programmable cut - off. The cut - off defines the end points of various segments (and, indirectly, the start point of the last segment). The output of the COMP block 809 is an identifier indicating which region (segment) the current input belongs to. This identifier is used to select the appropriate set of coefficients for approximating the non - linear function within that segment.

[0109]

[0115] In an exemplary implementation, 11 cutoff values are used to define a total of 12 segments. By utilizing the symmetry of the non-linear function, the COMP module operates only on the absolute value of the input. The following algorithm describes its operation.

[0110]

Number

[0111]

[0116] "TRUNC" represents a truncation block, which is configured to reduce the precision of a fraction represented in two's complement form by omitting the least significant bits. For example, to reduce the input from having M fractional bits to having N fractional bits, the least significant (M - N) bits are omitted (truncated).

[0112]

[0117] "SYMMSAT" represents a "symmetric saturation" block. A conventional saturation (SAT) module with N integer bits saturates at the largest positive value (2 N-1 - 1) or the smallest negative value (-2 N-1 ). In the symmetric saturation module, negative inputs instead saturate at a value greater than the smallest negative value (e.g., -2 N-1 + 1). In this way, the number of different negative and positive outputs is the same. For example, if [x > 0111...11], then y(0111...11) is output, and if [x <= 1000...00], then y(1000...01) is output.

[0113]

[0118] "SE" represents a sign extension block, which is useful when increasing the number of integer bits used to represent a value. For example, when attempting to represent a value represented by N integer bits with M integer bits (M > N), the sign bit is replicated (M - N) times.

[0114]

[0119] "TC" represents a block used to convert a positive number to a negative number by taking the two's complement.

[0115]

[0120] Finally, "ZF" represents a "zero-padding" block that is used by adding (M - N) zeros to the least significant position when a number with N fractional bits is required to be represented as a number with M fractional bits (where M > N).

[0116]

[0121] FIG. 9 shows an example of a linear approximation circuit 900 that can be an example of the linear approximation blocks 804A - 804C in the circuit 800 of FIG. 8. As shown, this linear approximation circuit includes a multiplication element 902 that receives a stage input and a coefficient 904 as inputs, and an adder element 906 that receives the output of the multiplication element after being processed by a truncation and saturation block and a second coefficient 908 as inputs.

[0117] Exemplary processing system

[0122] FIG. 10 shows an exemplary processing system 1000 that can be configured to implement the methods described herein, such as with respect to FIGS. 4 - 5.

[0118]

[0123] The processing system 1000 includes a central processing unit (CPU) 1002, which can be a multi-core CPU in some examples. Instructions executed in the CPU 1002 may be loaded, for example, from a program memory associated with the CPU 1002 or from a memory partition 1024.

[0119]

[0124] The processing system 1000 also includes additional processing components organized for specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, a neural processing unit (NPU) 1008, a multimedia processing unit 1010, and a wireless connectivity component 1012.

[0120]

[0125] For example, an NPU such as 1008 is a special circuit configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. The NPU may alternatively be referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligence processing unit (IPU), or vision processing unit (VPU).

[0121]

[0126] For example, an NPU such as 1008 can be configured to accelerate the performance of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs can be instantiated on a single chip such as a system-on-chip (SoC), while in other examples, they can be part of a dedicated machine learning accelerator device.

[0122]

[0127] The NPU can be optimized for training or inference, or, in some cases, configured to balance performance between the two. For NPUs capable of performing both training and inference, the two tasks can generally still be performed independently.

[0123]

[0128] An NPU designed to accelerate training is generally configured to accelerate the optimization of a new model, which involves inputting an existing dataset (often labeled or tagged), iterating through that dataset, and then adjusting model parameters such as weights and biases to improve model performance, which is a very computationally intensive operation. Generally, optimization based on incorrect predictions involves backpropagating through the layers of the model to determine gradients to reduce the prediction error.

[0124]

[0129] NPUs designed to accelerate inference are generally configured to operate on a complete model. Thus, such an NPU may be configured to rapidly process data through a model that has already been trained to take new data as input and produce a model output (e.g., an inference).

[0125]

[0130] In some embodiments, NPU 1008 may be implemented as part of one or more of CPU 1002, GPU 1004, and / or DSP 1006.

[0126]

[0131] In some embodiments, the wireless connectivity component 1012 may include sub-components for, e.g., third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 1012 is further connected to one or more antennas 1014.

[0127]

[0132] The processing system 1000 may also include one or more sensor processing units 1016 associated with any type of sensor, one or more image signal processors (ISPs) 1018 associated with any type of image sensor, and / or a navigation processor 1020 that may include satellite-based positioning system components (e.g., GPS or GLONASS), as well as inertial positioning system components.

[0128]

[0133] The processing system 1000 may also include one or more input and / or output devices 1022 such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, etc.

[0129]

[0134] In some examples, one or more of the processors of the processing system 1000 may be based on the ARM or RISC-V instruction set.

[0130]

[0135] The processing system 1000 also includes various circuits according to the various embodiments described herein.

[0131]

[0136] In this example, the processing system 1000 includes a compute-in-memory (CIM) circuit 1026 that can be configured to execute an efficient multiply-accumulate (MAC) function for processing machine learning model data. The processing system 1000 further includes a configurable non-linear activation (CNLA) function circuit 1028. In some cases, the CNLA function circuit 1028 may be similar to the CNLA function circuit 200 described with respect to FIGS. 2, 3, 6, 8, and 9. The CNLA function circuit 1028, as well as other circuits not shown, can be configured to execute various aspects of the methods described herein, such as the methods 400 and 500 associated with FIGS. 4 and 5, respectively.

[0132]

[0137] In some examples, the CNLA function circuit 1028 can be implemented as part of another processing unit, such as a CPU 1002, GPU 1004, DSP 1006, or NPU 1008.

[0133]

[0138] The processing system 1000 also includes a memory 1024 representing one or more static memories and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 1024 includes computer-executable components that can be executed by one or more of the above-described components of the processing system 1000.

[0134]

[0139] In particular, in this example, the memory 1024 includes a determination component 1024A, a setting component 1024B, a processing component 1024C, an extraction component 1024D, a non-linear activation function parameter 1024E, a lookup table(s) 1024F, and model parameters 1024G (e.g., weights, biases, and other machine learning model parameters). One or more of the illustrated components, as well as other components not shown, may be configured to implement various aspects of the methods described herein.

[0135]

[0140] Generally, the processing system 1000 and / or its components may be configured to implement the methods described herein.

[0136]

[0141] In particular, in other embodiments, aspects of the processing system 1000 may be omitted, such as when the processing system 1000 is a server computer or the like. For example, the multimedia component 1010, the wireless connectivity 1012, the sensor 1016, the ISP 1018, and / or the navigation component 1020 may be omitted in other embodiments. Further, aspects of the processing system 1000 may be distributed.

[0137]

[0142] Note that FIG. 10 is merely an example, and in other examples, alternative processing systems with more components, fewer components, and / or different components may be used.

[0138] Exemplary clauses

[0143] In the following numbered clauses, implementation examples are described.

[0139]

[0144] Clause 1: A processor comprising a configurable non - linear activation function circuit, which is configured to determine a non - linear activation function for applying to input data, determine a set of parameters for the non - linear activation function based on the determined non - linear activation function, and generate output data based on the application of the set of parameters for the non - linear activation function, the configurable non - linear activation function circuit.

[0140]

[0145] Clause 2: The configurable non - linear activation function circuit of Clause 1, comprising a first approximator configured to approximate a first function using one or more first function parameters of the set of parameters, a second approximator configured to approximate a second function using one or more second function parameters of the set of parameters, a gain multiplier configured to multiply a gain value based on one or more gain parameters of the set of parameters, and a constant adder configured to add a constant value based on a constant parameter of the set of parameters.

[0141]

[0146] Clause 3: The processor of Clause 2, wherein at least one of the first approximator and the second approximator is a cubic approximator.

[0142]

[0147] Clause 4: The processor of Clause 3, wherein the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator.

[0143]

[0148] Clause 5: The processor of Clause 2, wherein both the first approximator and the second approximator are cubic approximators.

[0144]

[0149] Clause 6: The processor of Clause 3, wherein the other of the first approximator and the second approximator is configured to obtain an approximation value and access a look - up table.

[0145]

[0150] Clause 7: The other of the first approximator and the second approximator is the processor according to clause 3, configured to execute a min / max function.

[0146]

[0151] Clause 8: The determined non-linear activation function includes a Swish function, the gain parameter includes a dependent parameter value of 1 and an independent parameter value of 0, the constant value is 0, the first function is a quadratic function, and the second function is a sigmoid lookup table, for the processor according to clause 2.

[0147]

[0152] Clause 9: The determined non-linear activation function includes a hard Swish function, the gain parameter includes a dependent parameter value of 1 / 6 and an independent parameter value of 0, the constant value is 3, the first function is a max function, and the second function is a min function, for the processor according to clause 2.

[0148]

[0153] Clause 10: The determined non-linear activation function includes a hyperbolic tangent (tanh) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a tanh lookup table, for the processor according to clause 2.

[0149]

[0154] Clause 11: The determined non-linear activation function includes a sigmoid function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is linear, and the second function is a sigmoid lookup table, for the processor according to clause 2.

[0150]

[0155] Clause 12: The determined non-linear activation function includes a Gaussian error linear unit (GELU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 1, the first function is a cubic function, and the second function is a tanh lookup table, for the processor according to clause 2.

[0151]

[0156] Clause 13: The determined non-linear activation function includes a rectified linear unit (ReLU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a max function, the processor described in Clause 2.

[0152]

[0157] Clause 14: The determined non-linear activation function includes a rectified linear unit - 6 (ReLU6) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a max function, and the second function is a min function, the processor described in Clause 2.

[0153]

[0158] Clause 15: The determined non-linear activation function includes an exponential linear unit (ELU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of α, the constant value is 0, the first function is a quadratic function when the input data value is ≥ 0, or is bypassed when the input data value is < 0, and the second function is bypassed when the input data value is ≥ 0, or is an exponential look-up table when the input data value is < 0, the processor described in Clause 2.

[0154]

[0159] Clause 16: An input memory buffer configured to store one or more outputs received from the processing circuit as input data, and an output memory buffer configured to store output data generated for the output of the configurable non-linear activation function circuit, the processor according to any one of Clauses 1 to 15.

[0155]

[0160] Clause 17: A compute-in-memory array configured to provide input data to a configurable non-linear activation function circuit, the processor according to any one of Clauses 1 to 16.

[0156]

[0161] Clause 18: A method for processing input data by a configurable non-linear activation function circuit, comprising: determining a non-linear activation function to apply to the input data; determining a set of parameters for the configurable non-linear activation function circuit based on the determined non-linear activation function; and processing the input data using the configurable non-linear activation function circuit based on the set of parameters to generate output data.

[0157]

[0162] Clause 19: The method according to Clause 18, further comprising retrieving a set of parameters from a memory based on the determined non-linear activation function.

[0158]

[0163] Clause 20: The method according to Clause 18, wherein the set of parameters includes a combination of one or more gain parameters, constant parameters, and one or more approximation functions for applying to the input data via the configurable non-linear activation function circuit.

[0159]

[0164] Clause 21: The configurable non-linear activation function circuit comprises: a first approximator configured to approximate a first function among one or more approximation functions; a second approximator configured to approximate a second function among one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value based on a constant parameter. The method according to Clause 20.

[0160]

[0165] Clause 22: The configurable non-linear activation function circuit further comprises: a first bypass configured to bypass the first approximator; a second bypass configured to bypass the second approximator; and an input data bypass configured to bypass the first approximator and provide the input data to the second approximator. The method according to Clause 21.

[0161]

[0166] Clause 23: The method according to clause 22, wherein at least one of the first approximator and the second approximator is a cubic approximator.

[0162]

[0167] Clause 24: The method according to clause 23, wherein the other of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator.

[0163]

[0168] Clause 25: The method according to item 23, wherein both the first approximator and the second approximator are cubic approximators.

[0164]

[0169] Clause 26: The method according to clause 23, wherein the other of the first approximator and the second approximator is configured to obtain an approximation value and access a look-up table.

[0165]

[0170] Clause 27: The method according to clause 23, wherein the other of the first approximator and the second approximator is configured to execute a min / max function.

[0166]

[0171] Clause 28: The method according to clause 21, wherein the determined non-linear activation function includes a Swish function, the gain parameter includes a dependent parameter value of 1 and an independent parameter value of 0, the constant value is 0, the first function is a quadratic function, and the second function is a sigmoid look-up table.

[0167]

[0172] Clause 29: The method according to clause 21, wherein the determined non-linear activation function includes a Hard-Swish function, the gain parameter includes a dependent parameter value of 1 / 6 and an independent parameter value of 0, the constant value is 3, the first function is a max function, and the second function is a min function.

[0168]

[0173] Clause 30: The method according to clause 21, wherein the determined non-linear activation function includes a Gaussian Error Linear Unit (GELU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 1, the first function is a cubic function, and the second function is a tanh look-up table.

[0169]

[0174] Clause 31: The determined non-linear activation function includes the hyperbolic tangent (tanh) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is a tanh lookup table, the method according to Clause 21.

[0170]

[0175] Clause 32: The determined non-linear activation function includes the sigmoid function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a linear function, and the second function is a sigmoid lookup table, the method according to Clause 21.

[0171]

[0176] Clause 33: The determined non-linear activation function includes the rectified linear unit (ReLU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is a quadratic function, and the second function is the max function, the method according to Clause 21.

[0172]

[0177] Clause 34: The determined non-linear activation function includes the rectified linear unit - 6 (ReLU6) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is the max function, and the second function is the min function, the method according to Item 21.

[0173]

[0178] Clause 35: The determined non-linear activation function includes the exponential linear unit (ELU) function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of α, the constant value is 0, the first function is a quadratic function when the input data value is ≧ 0, or is bypassed when the input data value is < 0, and the second function is bypassed when the input data value is ≧ 0, or is an exponential lookup table when the input data value is < 0, the method according to Clause 21.

[0174]

[0179] The method according to clause 21, further comprising receiving input data from a compute-in-memory (CIM) array.

[0175]

[0180] Clause 37: A configurable non-linear activation function circuit configured to process a non-linear activation function according to any of the settings in Table 1.

[0176]

[0181] Clause 38: A circuit block comprising a configurable non-linear activation function circuit and a selectable bypass.

[0177]

[0182] Clause 39: The circuit block according to clause 38, wherein the configurable non-linear activation function circuit is configured to process a non-linear activation function according to any of the settings in Table 1.

[0178]

[0183] Clause 40: The circuit block according to clause 38, wherein the circuit block can be configured according to any of the settings in Table 2.

[0179]

[0184] Clause 41: A processor comprising a configurable non-linear activation function circuit configured to determine a set of parameters for a selected non-linear activation function based on the selected non-linear activation function and to generate output data based on the application of the set of parameters for the selected non-linear activation function, wherein the configurable non-linear activation function circuit comprises at least one non-linear approximator comprising at least two sequential linear approximators, and each linear approximator of the at least two sequential linear approximators is configured to approximate a linear function using one or more function parameters of the set of parameters.

[0180]

[0185] Clause 42: The processor according to clause 41, wherein each linear approximator of the at least two sequential linear approximators comprises a stage input, a coefficient input, a constant input, and a stage output.

[0181]

[0186] Clause 43: The processor according to any one of Clauses 41 to 42, wherein at least one non-linear approximator comprises a cubic approximator including three sequential linear approximators.

[0182]

[0187] Clause 44: The processor according to any one of Clauses 41 to 43, wherein at least one non-linear approximator comprises a quadratic approximator including two sequential linear approximators.

[0183]

[0188] Clause 45: The processor according to Clause 42, wherein at least one non-linear approximator further comprises a region finder component configured to determine an input value region, and each linear approximator of at least two sequential linear approximators is further configured to determine a coefficient input and a constant input based on the input value region.

[0184]

[0189] Clause 46: The processor according to any one of Clauses 41 to 45, wherein at least one non-linear approximator further comprises a sign / offset corrector component configured to correct a stage output from at least one linear approximator of at least two sequential linear approximators.

[0185]

[0190] Clause 47: The processor according to Clause 46, wherein the sign / offset corrector component is further configured to invert the sign of the stage output to correct the stage output from at least one linear approximator of at least two sequential linear approximators.

[0186]

[0191] Clause 48: The processor according to Clause 46, wherein the sign / offset corrector component is further configured to add an offset to the stage output to correct the stage output from at least one linear approximator of at least two sequential linear approximators.

[0187]

[0192] Item 49: Each linear approximator of at least two sequential linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of function approximation for the linear approximator of at least two sequential linear approximators to approximate a linear function using one or more of the function parameters of a set of parameters, the processor according to any one of clauses 41 to 48.

[0188]

[0193] Item 50: A method for processing data using a configurable non-linear activation function circuit, comprising: determining a set of parameters for a selected non-linear activation function based on the selected non-linear activation function; and generating output data based on application of the set of parameters for the selected non-linear activation function, wherein the configurable non-linear activation function circuit comprises at least one non-linear approximator including at least two sequential linear approximators, and each linear approximator of the at least two sequential linear approximators is configured to approximate a linear function using one or more of the function parameters of a set of parameters.

[0189]

[0194] Item 51: Each linear approximator of at least two sequential linear approximators comprises a stage input, a coefficient input, a constant input, and a stage output, the method according to item 50.

[0190]

[0195] Item 52: The at least one non-linear approximator comprises a cubic approximator including three sequential linear approximators, the method according to any one of items 50 or 51.

[0191]

[0196] Item 53: The at least one non-linear approximator comprises a quadratic approximator including two sequential linear approximators, the method according to any one of items 50 to 52.

[0192]

[0197] Clause 54: The method according to clause 51, wherein at least one non-linear approximator further comprises a region discovery component configured to determine an input value region, and each linear approximator of at least two sequential linear approximators is further configured to determine a coefficient input and a constant input based on the input value region.

[0193]

[0198] Clause 55: The method according to any one of clauses 50 - 54, further comprising using a sign-offset corrector component to correct the stage output from at least one linear approximator of at least two sequential linear approximators.

[0194]

[0199] Clause 56: The method according to clause 55, further comprising using a sign-offset corrector component to invert the sign of the stage output to correct the stage output from at least one linear approximator of at least two sequential linear approximators.

[0195]

[0200] Clause 57: The method according to clause 55, further comprising using a sign-offset corrector component to add an offset to the stage output to correct the stage output from at least one linear approximator of at least two sequential linear approximators.

[0196]

[0201] Clause 58: The method according to any one of clauses 50 - 57, wherein each linear approximator of at least two sequential linear approximators is further configured to select one or more function parameters based at least in part on selecting one or more non-uniform segments of function approximation for the linear approximator of at least two sequential linear approximators to approximate a linear function using one or more of the set of parameters.

[0197]

[0202] Article 59: A processing system comprising a memory containing computer-executable instructions and one or more processors configured to execute the computer-executable instructions to cause the processing system to perform the method described in any one of Articles 18 to 36 or Articles 50 to 58.

[0198]

[0203] Article 60: A processing system comprising means for performing the method described in any one of Articles 18 to 36 or 50 to 58.

[0199]

[0204] Article 61: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method described in any one of Articles 18 to 36 or 50 to 58.

[0200]

[0205] Article 62: A computer program product embodied on a computer-readable storage medium comprising code for performing the method described in any one of Articles 18 to 36 or 50 to 58.

[0201] Additional Considerations

[0206] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples described herein are not intended to limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, the functions and arrangements of the elements described may be changed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as necessary. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, the features described for some embodiments may be combined with those of some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects described herein. In addition, the scope of the disclosure is intended to include devices or methods practiced using other structures, functions, or a combination of structures and functions in addition to, or other than, the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein can be embodied by one or more elements of the claims.

[0202]

[0207] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other aspects.

[0203]

[0208] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single member. By way of example, "at least one of a, b, or c" is intended to include a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination having multiple identical elements (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c, or any other order of a, b, and c).

[0204]

[0209] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), ascertaining, etc. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Further, "determining" may include solving, selecting, choosing, establishing, etc.

[0205]

[0210] As used herein, the term "connected to" may generally have meaning in the context of data communication between corresponding elements connected to each other in a context where electronic signals and data are shared between elements described herein. In some cases, elements may be directly connected to each other via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between corresponding elements directly connected to each other, for example. In other cases, elements may be indirectly connected to each other via one or more data buses or similar shared circuits and / or integrated circuit elements for communicating signals and data between corresponding elements indirectly connected to each other, for example.

[0206]

[0211] The methods disclosed in this specification include one or more steps or actions for achieving the methods. The steps and / or actions of those methods can be interchanged with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order of specific steps and / or actions, and / or the use of those steps and / or actions, can be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above can be implemented by any suitable means capable of performing the corresponding functions. Those means can include various hardware components and / or software components (singular or plural), and / or various hardware modules and / or software modules (singular or plural), including but not limited to circuits, application specific integrated circuits (ASICs), or processors. Generally, when operations are shown in the figures, those operations can have corresponding equivalent means-plus-function components with similar numbers.

[0207]

[0212] The following claims are not intended to be limited to the embodiments shown in this specification, but rather should be given the full scope consistent with the language of the claims. In the claims, reference to an element in the singular is not intended to mean "one and only one" unless explicitly so stated, but rather is intended to mean "one or more." Unless otherwise specified, the term "some" refers to one or more. An element of a claim should not be construed under 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, the element is expressly recited using the phrase "step of." All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.

Claims

1. It is a processor, A configurable nonlinear activation function circuit, Based on the selected nonlinear activation function, determine the set of parameters for the selected nonlinear activation function, A configurable nonlinear activation function circuit is configured to generate output data based on the application of the set of parameters for the selected nonlinear activation function, The configurable nonlinear activation function circuit comprises at least one nonlinear approximator including at least two successive linear approximators, Each of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters from the set of parameters, Processor.

2. Each of the at least two successive linear approximators is: Stage input and, Coefficient input and, Constant input and, Stage output and, Equipped with, The processor according to claim 1.

3. At least one nonlinear approximator comprises a cubic approximator including three successive linear approximators, or The processor according to claim 1, wherein the at least one nonlinear approximator comprises a quadratic approximator including two successive linear approximators.

4. The at least one nonlinear approximator further comprises a region finder component configured to determine the input value region, Each of the at least two successive linear approximators is further configured to determine the coefficient input and the constant input based on the input value region. The processor according to claim 2.

5. The processor according to claim 1, wherein the at least one nonlinear approximator further comprises a sign-offset corrector component configured to correct the stage output from at least one linear approximator among the at least two successive linear approximators.

6. The sign / offset corrector component is further configured to invert the sign of the stage output in order to correct the stage output from at least one of the at least two successive linear approximators, or The processor according to claim 5, wherein the sign / offset corrector component is further configured to add an offset to the stage output in order to correct the stage output from at least one linear approximator among the at least two successive linear approximators.

7. The processor according to claim 1, wherein each linear approximator of the at least two successive linear approximators is further configured to select the one or more function parameters, at least in part on the basis that each linear approximator of the at least two successive linear approximators selects one or more heterogeneous segments of a function approximation in order to approximate a linear function using one or more function parameters from the set of parameters.

8. A method for processing data using a configurable nonlinear activation function circuit, Based on the selected nonlinear activation function, determine the set of parameters for the selected nonlinear activation function, This includes generating output data based on the application of the set of parameters for the selected nonlinear activation function, The configurable nonlinear activation function circuit comprises at least one nonlinear approximator including at least two successive linear approximators, Each of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters from the set of parameters, method.

9. Each of the at least two successive linear approximators is: Stage input and, Coefficient input and, Constant input and, Stage output and, Equipped with, The method according to claim 8.

10. The method according to claim 8, wherein at least one nonlinear approximator comprises a cubic approximator including three successive linear approximators.

11. The method according to claim 8, wherein the at least one nonlinear approximator comprises a quadratic approximator including two successive linear approximators.

12. The at least one nonlinear approximator further comprises a region finder component configured to determine the input value region, Each of the at least two successive linear approximators is further configured to determine the coefficient input and the constant input based on the input value region. The method according to claim 9.

13. The method according to claim 8, further comprising using a sign / offset corrector component to correct the stage output from at least one linear approximator among the at least two successive linear approximators.

14. To correct the stage output from at least one of the two successive linear approximators, the sign / offset corrector component is used to invert the sign of the stage output, or To modify the stage output from at least one of the two successive linear approximators, the sign / offset corrector component is used to add an offset to the stage output. The method according to claim 13, further comprising:

15. The method according to claim 8, wherein each of the at least two successive linear approximators is further configured to select the one or more function parameters, at least in part, on the basis of selecting one or more heterogeneous segments of a function approximation in order to approximate a linear function using one or more function parameters from the set of parameters.

16. A non-temporary computer-readable medium comprising computer-executable instructions, wherein when the computer-executable instructions are executed by a processor comprising a configurable nonlinear activation function circuit, the processor is caused to execute the method according to any one of claims 8 to 15.

Images