configurable nonlinear activation function circuit

Abstract

Some aspects of the present disclosure provide a method for processing input data by a set of configurable nonlinear activation function circuits, the method comprising: generating an exponential output by processing the input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function; adding the exponential outputs of one or more first configurable nonlinear activation function circuits; and generating an approximate logarithmic softmax output by processing the added exponential outputs using a second configurable nonlinear activation function circuit configured to perform a natural logarithmic function.

Description

[Technical Field]

[0001] (Cross-reference of related cases) This application claims priority to U.S. Patent Application No. 18 / 165,802, filed on February 7, 2023, which is a continuation-in-part application of U.S. Patent Application No. 17 / 467,079, filed on September 3, 2021, the entirety of which is incorporated herein by reference.

[0002] introduction Aspects of this disclosure relate to processing nonlinear activation functions for machine learning models, and more particularly to configurable nonlinear activation function circuits.

[0003] Machine learning is, generally speaking, the process of producing a trained model (e.g., an artificial neural network), which represents a generalized fit to a set of training data. Applying the trained model to new data makes it possible to produce inferences, which can be used to gain insights into the new data.

[0004] As the use of machine learning (or artificial intelligence) has surged to enable a variety of machine learning (or artificial intelligence) tasks, the need for more efficient processing of machine learning model data has arisen. In some cases, dedicated hardware such as machine learning (or artificial intelligence) accelerators or processors or similar circuits can be used to enhance the processing system's ability to handle machine learning model data. For example, processing data with nonlinear activation functions may be distributed to processors other than the first-order matrix multiplication processor. However, distributing various aspects of processing machine learning models across different processing devices can result in latency, memory usage, power consumption, and other processing penalties.

[0005] Therefore, improved techniques are needed for processing machine learning model data using nonlinear activation functions. [Overview of the project]

[0006] Some embodiments provide a processor comprising: one or more first configurable nonlinear activation function circuits configured to perform an exponential function on input data; an adder circuit configured to receive output data from one or more first configurable nonlinear activation function circuits; and a second configurable nonlinear activation function circuit configured to receive output data from the adder circuit, perform a natural logarithmic function, and output an approximate logarithmic softmax of the input data.

[0007] A further embodiment provides a method for processing input data with a set of configurable nonlinear activation function circuits, comprising: generating an exponential output by processing the input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function; adding the exponential outputs of one or more first configurable nonlinear activation function circuits; and generating an approximate logarithmic softmax output by processing the added exponential outputs using a second configurable nonlinear activation function circuit configured to perform a natural logarithmic function.

[0008] Other embodiments provide a processing system configured to carry out the aforementioned method and the methods described herein; a non-temporary computer-readable medium containing instructions that, when executed by one or more processors of the processing system, cause the processing system to carry out the aforementioned method and the methods described herein; a computer program product embodied on a computer-readable storage medium comprising code that carries out the aforementioned method and the methods further described herein; and a processing system comprising means for carrying out the aforementioned method and the methods further described herein.

[0009] The following description and related drawings provide a detailed representation of specific exemplary features of one or more embodiments.

[0010] The accompanying drawings illustrate some of the one or more aspects and thus should not be considered as limiting the scope of the disclosure.

Brief Description of the Drawings

[0011] [Figure 1] An exemplary configurable nonlinear activation (CNLA) function circuit is shown. [Figure 2] An exemplary circuit block for implementing a bypassable approximator block as described with respect to FIG. 1 is shown. [Figure 3] An exemplary approximator is shown. [Figure 4] An exemplary process flow of a machine learning model is shown. [Figure 5] An exemplary method for performing processing using a configurable nonlinear activation function circuit is shown. [Figure 6] An exemplary architecture of a CNLA function circuit for performing a softmax operation using parallel input data is shown. [Figure 7] An exemplary architecture of a CNLA function circuit for performing a softmax operation using sequential input data is shown. [Figure 8] An exemplary method for performing an approximate softmax operation using a CNLA function circuit is shown. [Figure 9] An exemplary processing system configured to perform the methods described herein is shown.

[0012] For ease of understanding, where possible, the same reference numbers are used to designate the same elements common to the drawings. It is contemplated that elements and features of one aspect can be beneficially incorporated into other aspects without further elaboration.

Modes for Carrying Out the Invention

[0013] Aspects of this disclosure provide improved techniques for handling nonlinear activation functions associated with machine learning models.

[0014] Nonlinear activation is a crucial component of various types of machine learning models, including neural network models. Some nonlinear activation functions are implemented as piecewise linear functions (e.g., ReLU (Normalized Linear Unit), Leaky ReLU, etc.), while others require complex mathematical functions (e.g., sigmoid, hyperbolic tangent (tanh), etc.). In some cases, complex mathematical functions may be implemented using interpolation, such as cubic spline interpolation. For example, the interpolated output value may, in some embodiments, be determined using a lookup table (LUT) to match the output value to the input value. When the target input value is not mapped in the LUT, LUT values ​​associated with input values ​​adjacent to the target input value may be used to interpolate the output value for the target input value.

[0015] Traditionally, nonlinear activation functions have been implemented in software rather than hardware due to the wide range of possible activation functions available in machine learning models. However, such implementations typically require moving model data between processing devices (e.g., between a neural processing unit (NPU) that performs matrix multiplication and accumulation and a digital signal processor (DSP) that processes the nonlinear activation functions), thus incurring power and latency penalties. When nonlinear activation functions are implemented in hardware, they are generally limited to supporting only a small number of nonlinear activation functions and therefore cannot be configured to support evolving machine learning model architectures without delegating the processing of nonlinear activation functions to other processing units.

[0016] For example, the Normalized Linear Unit (ReLU) is a commonly used activation function in deep learning models. This function returns 0 if it receives a negative input, and otherwise returns the input x. Thus, it can be written as f(x) = max(0,x). The ReLU function is generally not implemented by a linear matrix multiplication and accumulation processing unit such as a compute-in-memory (CIM) array, in some examples. Therefore, the need to distribute the ReLU function or other nonlinear activation functions can be costly from a processing perspective. Furthermore, as the activation function becomes more complex, the processing cost (for example, to perform relatively higher power exponential and division operations, which are some of the nonlinear activation functions, as will be discussed further below) also becomes more significant.

[0017] To overcome the shortcomings of conventional solutions, the embodiments described herein relate to a configurable nonlinear activation (CNLA) function circuit that can be implemented in hardware for efficient processing. In particular, since the CNLA function circuit can be implemented in hardware, the CNLA function can be placed in the same location as other processing circuits optimized for other machine learning model processing tasks, such as CIM arrays and digital multiply-and-accumulate (DMAC) circuits optimized to perform vector and matrix multiplication and accumulation functions.

[0018] To improve processing efficiency, the embodiments described herein may use polynomial approximations to approximate complex functions, such as those that may be used within nonlinear activation functions. In some cases, the embodiments described herein may use series expansions, such as Taylor series. Generally, the Taylor series of a function (e.g., f(x)) is an infinite sum of terms expressed with respect to the derivative of the function at a single point. For many functions, the sum of the function and its Taylor series is equal near this point. The partial sum formed by the first n+1 terms of the Taylor series is a polynomial of degree n called the nth-degree Taylor polynomial of the function. Thus, Taylor polynomials enable efficient approximations of functions, which generally improve as n increases.

[0019] The CNLA function circuits described herein can generally implement one or more polynomial approximation blocks, such as cubic approximation blocks, which improve cubic spline interpolation to be more efficient, more generalizable, and cover a wider variety of nonlinear activation functions. Furthermore, the CNLA function circuits may be implemented as pipelining digital blocks that can use nonlinearly segmented look-up tables (LUTs) and mixed-order approximations (e.g., pipelining linear, quadratic, and cubic approximations). Thus, unlike conventional nonlinear activation function circuits, the CNLA function circuits described herein can be configured to meet many different performance objectives.

[0020] Therefore, the CNLA function circuits described herein provide a technical solution to the technical problem of implementing a wide range of nonlinear activation functions in machine learning model processing systems. Furthermore, the CNLA function circuits described herein offer technical improvements, including lower latency, lower power consumption, improved memory efficiency, and other advantages described herein, by improving model processing performance compared to existing solutions.

[0021] Exemplary configurable nonlinear activation function circuits Figure 1 shows an exemplary configurable nonlinear activation (CNLA) function circuit 100.

[0022] Generally, the CNLA function circuit 100 may be configured to receive input data 101 (e.g., output values ​​from layers of a machine learning model) and execute various nonlinear activation functions to generate output data 114 (e.g., "activation"). The CNLA function circuit 100 may be located in the same location as other machine learning model processing circuits such as CIM arrays and DMACs, and may be pipelined, and may be configured to execute activation functions based on the outputs of other machine learning model processing circuits.

[0023] In some examples, input data 101 may be received from a buffer or other memory. In other examples, input data 101 may be received directly from the output of another processing block, such as a CIM array or the output of another vector and matrix multiplication / accumulation block.

[0024] The CNLA function circuit 100 includes a first approximator block 102, which can generally be configured to perform hardware-based mathematical functions on input data 101, etc. An exemplary approximator is described in detail with respect to Figure 3.

[0025] In some cases, the first approximator is a linear approximator (configured to perform linear functions such as ax+b, for example, ax 2 A quadratic approximator configured to perform quadratic functions such as ax + bx + c, (e.g., ax 3 +bx 2This is one of the cubic approximators (configured to perform a cubic function such as +cx+d), where x is the input data and a, b, c, and d are configurable parameters. In general, a linear, quadratic, or cubic approximator can be used to approximate any given function (which may or may not be a polynomial function). The first approximator 102 can be configured by parameters retrieved from, for example, memory, registers, or lookup tables. Many types of nonlinear activation functions can be approximated using these different forms of approximation and associated configurable parameters, as will be described in more detail below with respect to Table 2.

[0026] The CNLA function circuit 100 further includes a second approximator block 104, which, like the first approximator block 102, can generally be configured to perform hardware-based mathematical functions such as linear, quadratic, or cubic functions. As will be described in more detail below, the CNLA function circuit 100 can be configured to use the first approximator block 102 and the second approximator block 104 in series for more complex functions, such that the output of the first approximator block 102 becomes the input to the second approximator block 104. The CNLA function circuit 100 may further be configured to use only one of the first or second approximator block 102 or 104 when simpler nonlinear functions are being handled, thereby saving power.

[0027] In some implementations, the first approximator 102 and the second approximator 104 may comprise the same circuit block (e.g., two instances of the same circuit element within circuit 100). For example, the first approximator 102 and the second approximator 104 may comprise a cubic approximator in some embodiments. In other implementations, the first approximator 102 and the second approximator 104 may comprise different circuit elements, in which case, generally, the second approximator 104 comprises a cubic approximator and the first approximator 102 comprises a lower-order approximator, such as a quadratic or linear approximator. However, in other embodiments, the order of higher-order and lower-order approximators may be reversed.

[0028] The CNLA function circuit 100 includes a configurable bypass 105 that allows the first approximator 102 to be bypassed in various scenarios, such as when the function requires only an approximator of a lower order than the first approximator 102, and the second approximator 104 is such a lower order approximator. For example, when the first approximator 102 is bypassed via the configurable bypass 105, the input data 101 is instead provided directly to the second approximator 104 and not processed by the first approximator 102. In various embodiments, the first approximator 102 may be a higher-order approximator than the second approximator 104, or vice versa, or they may be of the same order (e.g., both linear, quadratic, or cubic). The configurable bypass 105 allows for saving processing time and energy when only one approximator is required.

[0029] The CNLA function circuit 100 further includes another configurable bypass 107, which allows the second approximator 104 to be bypassed in various scenarios, such as when the function requires only a first approximation, and the first approximator 102 can run without the second approximator 104. For example, when the second approximator 104 is bypassed via the configurable bypass 107, the output of the first approximator 102 is provided directly to the multiplier 108.

[0030] In general, the configurable bypasses 105 and 107 allow the CNLA function circuit 100 to be configured for maximum versatility while saving power and avoiding unnecessary circuit blocking in various scenarios. Furthermore, the configurable bypasses allow asymmetric and antisymmetric nonlinear activation functions to be configured to be handled by the CNLA function circuit 100. Figure 2 shows exemplary circuit configurations for implementing the configurable bypasses 105 and 107 (e.g., bypasses 205A and 205B).

[0031] The CNLA function circuit 100 further includes a gain block 106 configured to provide a gain value to the multiplier 108. In some embodiments, the gain block 106 is configured to generate a gain value 109 based on a gain function implemented by the gain block 106. In one example, the gain function may be of the form g = ax + b, where g is the gain value, x is the value of the input data 101, and a and b are configurable parameters. More generally, the gain block 106 can generate a gain value by multiplicatively (a) and / or additively (b) modifying the input data.

[0032] The gain value 109 generated by the gain block 106 is multiplied by the outputs of the first approximator 102 and / or the second approximator 104 via the multiplier 108. In other embodiments, the gain block 106 may be constructed using a gain value that is not based on a function of the input data 101 (for example, by setting a to 0 in the above equation for g). In general, the parameters (e.g., a and b in the above example) or values ​​of the gain block 106 may be obtained from, for example, memory, registers, lookup tables, etc.

[0033] The CNLA function circuit 100 further includes a constant block 110 configured to store a configurable (e.g., programmable) constant value 113, and an adder 112 configured to add the constant value 113 to the output of a multiplier 108 (e.g., a gain multiplier). The constant value 113 stored in the constant block 110 can be retrieved from, for example, memory, registers, a lookup table, etc.

[0034] The inclusion and arrangement of the first approximator block 102, the second approximator block 104, configurable bypasses 105 and 107, the gain block 106, the multiplier 108, the constant block 110, and the adder 112 allows the CNLA function circuit 100 to be configured to execute a wide variety of known and future-developed nonlinear activation functions. Furthermore, the CNLA function circuit 100 can be efficiently configured to handle a wide variety of nonlinear activation functions simply by updating the parameters of the first approximator 102, the second approximator 104, the gain block 106, and the constant block 110. When both approximator blocks 102 and 104 are used together to simulate a nonlinear function, each approximator block 102 and 104 can be said to execute the corresponding individual functions (e.g., the first function executed by the first approximator block 102 and the second function executed by the second approximator 104). This design favorably supports arbitrary asymmetric nonlinear curves for complex functions.

[0035] Table 1 below provides exemplary parameters for various nonlinear activation functions that the CNLA function circuit 100 in Figure 1 may be configured to perform, including parameters for approximator blocks 206A and 206B in Figure 2. In Table 1, the gain is assumed to have the form ax+b, as in the example of the gain block 106 in Figure 1, but it should be noted that in other embodiments, the gain may be a scalar value or a different functional form. Similarly, the quadratic approximator is ax 2 It is assumed to have the form +bx+c, and the cubic approximator is ax 3 +bx 2 It is assumed to have the form +cx+d. In the table below, subscripts are used to indicate parameter assignments, for example, G for the gain parameter, 1 for the first approximator, and 2 for the second approximator parameter.

[0036] [Table 1] [Table 2] [Table 3]

[0037] Note that in the ELU function described above, the α parameter may be set as a hyperparameter by the model designer.

[0038] In particular, in some implementations, even when the approximator is performing a lower-order function (e.g., linear), the parameters of the approximator can be given in a formal form (e.g., cubic with parameters a, b, c, and d, or quadratic with parameters a, b, and c). This is because, for example, setting the cubic parameter a to 0 effectively collapses the approximation into a lower-order quadratic function, and similarly, setting the quadratic parameter a to 0 effectively collapses the approximation into a linear equation. Thus, the approximator can be set up for a "quadratic function" when set up with quadratic parameters, but the parameter result can reduce the function to a linear function, as in the ReLU example in Table 2 above. This allows for the standardization of the parameter set regardless of the order of the underlying function set by the parameters, thereby simplifying the implementation.

[0039] Figure 2 shows exemplary circuit blocks 202 and 204 for implementing bypassable approximator blocks 206A and 206B. Bypassable approximator blocks 206A and 206B may, in one example, correspond to the first approximator block 102 and the second approximator block 104 in Figure 1.

[0040] In Figure 2, circuit block 202 is configured to control the use of function block 214A, which in this example includes a first approximator 206A and a minimum and maximum function block 208A. Similarly, circuit block 204 controls the use of function block 214B, which in this example includes a minimum and maximum function block 208B and a second approximator 206B. The first and second approximator blocks 206A and 206B may be configured to implement nonlinear activation functions, such as those described above with respect to Table 1.

[0041] Note that while the first approximator 102 in Figure 1 requires only one input, circuit block 202 includes two input ports 201A and 201B, which allows for multiple inputs. The illustrated configuration of circuit block 202 can be adopted to present the same external interface to both circuit blocks 202 and 204, which may simplify configuration and integration. In some embodiments, the two input ports 201A and 201B of circuit block 202 can be coupled to each other in an implementation where circuit block 202 receives a single input (such as input data 101 in Figure 1) via input port 201A. In an alternative implementation, circuit block 202 can be simplified by removing input port 201B and input mux 203A, so that 201A is directly provided to 214A and 207A.

[0042] In general, input ports 201A and 201B can accept various types of input data for processing, including signed multi-bit integer data. For example, the input data may be 8-bit two's complement input data.

[0043] The input selectors mux203A and 203B are configured to control which input data ports are used for circuit blocks 202 and 204, respectively. For example, input selector mux203B can be selected between input data port 201A (e.g., when circuit block 202 is bypassed) or 212B (e.g., when circuit blocks 202 and 204 are processed in series).

[0044] Bypass selectors mux211A and 211B are configured to control bypass function blocks 214A and 214B of circuit blocks 202 and 204, respectively. For example, when circuit block 202 is bypassed, bypass selector mux211A selects bypass 205A to provide output to output port 212A. Similarly, when circuit block 204 is bypassed, bypass selector mux211B selects bypass 205B to provide output to output port 216. In this way, processing using circuit blocks 202 and / or 204 controlled by configurable bypasses 205A and 205B results in output at output port 216.

[0045] As will be explained in more detail with respect to Figure 3, approximator blocks 206A and 206B may be configured using setting parameters (e.g., function eigencoefficients as shown in Table 1 above) stored in registers 219A and 219B, respectively. Similarly, if approximator block 206A or 206B is configured to perform lookup table-based functionality as shown in Table 1 above, the table values ​​may be stored in registers 219A and 219B, respectively.

[0046] Each circuit block (202 and 204) further includes a minimum / maximum function block (208A for circuit block 202, and 208B for circuit block 204) to provide minimum and maximum functions. Generally, the minimum (or "min") function returns the smallest value among the provided inputs. Similarly, the maximum (or "max") function returns the largest value among the provided inputs. In one example, the minimum / maximum function blocks 208A and 208B may comprise a multi-bit digital comparator operating in either single-cycle or multi-cycle mode.

[0047] The settings for function blocks 214A and 214B may include settings for function selectors mux209A and 209B, respectively. In other words, whether function blocks 214A and 214B output min / max outputs from mix / max blocks 208A and 208B, or values ​​from approximators 206A and 206B, depends on the settings for function selectors mux209A and 209B. Note that in other examples, function blocks 214A and 214B may include additional function blocks that can be selected by mux.

[0048] As shown in Figure 1, when the approximator blocks can be processed in series, in Figure 2, the output 212A of circuit block 202A, which includes the first approximator block 206A, is provided as input 212B to circuit block 204, which includes the second approximator block 206B. Similar to Figure 1, where bypasses 105 and 107 control the use of the first and second approximator blocks 102 and 104, here, selectable bypasses 205A and 205B control the use of approximator blocks 206A and 206B.

[0049] The asymmetric signal line 210 controls the configuration of circuit blocks 202 and 204. For example, circuit blocks 202 and 204 are configured based on the value on the asymmetric signal line 210 and the output values ​​from code blocks 207A and 207B based on input data received via input data port 201A. For instance, the binary value received via the asymmetric signal line 210 and the binary value output from code block 207A interact at AND gate 213 to control the selection of the output mux211A. In another example, the binary value received via the asymmetric signal line 210 and the binary value output from code block 207B interact at AND gate 217 to control the selection of the input data port (such as between 201A and 212B) via mux203B. As yet another example, the binary value received via the asymmetric signal line 210 and the inverted binary value output from code block 207B interact at gate 215 to control the selection of the output mux211B.

[0050] Table 2 below provides an overview of the configurations for circuit blocks 202 and 204.

[0051] [Table 4]

[0052] Exemplary approximator for configurable nonlinear activation function circuits Figure 3 shows an exemplary approximator 300, which may be an example of one or both of the first approximator 102 and the second approximator 104 in Figure 1, and / or the approximators 206A and 206B in Figure 2.

[0053] The approximator 300 receives input data 302 (e.g., pre-activation data) for processing. In some examples, the input data 302 may be received from a buffer or other memory. In other examples, the input data may be received directly from the output of another processing block, such as the output of a CIM array or another vector and matrix multiplication / accumulation block. Furthermore, the input data may be received from another approximator, such as when the approximator 300 is the second approximator 104 in Figure 1 and / or the second approximator 206B in Figure 2.

[0054] In some implementations, the approximator (e.g., 300) may include alternative processing paths. In such cases, the routing logic 304 may be configured to route the input data 302 to the appropriate processing path, for example, based on configuration parameters for the approximator 300.

[0055] In this example, processing path 306A provides a cubic approximation path for the input data 302.

[0056] In processing path 306A, input data 302 is supplied to tertiary computer 308, and the tertiary computer performs tertiary calculations (for example, x 3 (where x is the input data) is executed, and then the output is multiplied by the cubic parameter 312 in the multiplier 310. The output of the multiplier 310 is then fed to the accumulator 324.

[0057] The input data 302 is also provided to the secondary calculator 314, which performs a quadratic operation (for example, x 2 (where x is the input data) is executed, and the output is then multiplied by the quadratic parameter 318 in the multiplier 316. The output of the multiplier 316 is then provided to the accumulator 324.

[0058] The input data 302 is also supplied to the multiplier 320, where it is multiplied by the linear parameter 322. The output of the multiplier 320 is then supplied to the accumulator 324.

[0059] The adder 324 accumulates the outputs of the multipliers 310, 316, and 320, as well as the intercept parameter 326, to generate output data 332.

[0060] The cubic parameter 312, the quadratic parameter 318, the linear parameter 322, and the intercept parameter 326 can all be stored in memory accessible from the approximator 300 (for example, in a register). In some cases, a control unit such as a memory control unit or a finite state machine can set the approximator 300 using the parameters stored in memory. In various examples, the cubic parameter 312, the quadratic parameter 318, the linear parameter 322, and the intercept parameter 326 may be set according to the values ​​described above with respect to Table 2.

[0061] In this way, the order of the approximation can be set by setting the parameter values ​​described above. For example, the cubic parameter 312 can be set to 0 so that the approximator 300 performs a quadratic approximation. Similarly, the cubic parameter 312 and the quadratic parameter 318 can be set to 0 so that the approximator 300 performs a linear approximation.

[0062] Certain nonlinear activation functions require alternative functions such as minimum and maximum functions. Therefore, processing path 306B provides minimum and / or maximum calculators that can be used, for example, with the ReLU and ReLU6 functions described above in Table 2. Processing path 306B may be selected by path logic 304 based on the setting data of approximator 300.

[0063] Furthermore, certain nonlinear activation functions may be implemented using lookup tables, which provide a more power-efficient and time-efficient mechanism for generating values ​​for specific nonlinear activation functions. Therefore, processing path 306C provides a lookup table-based processing path that can be used, for example, when a sigmoid function, a tanh function, or a similar function is used by the nonlinear activation function. It should be noted that sigmoid and tanh can be computed from each other, and therefore, in some cases, only a single lookup table (e.g., sigmoid or tanh, but not both) is stored and used to implement both functions. One or more lookup tables may be stored in memory and accessible from the approximator 300, including memory tightly coupled to the approximator 300.

[0064] Process flow of an exemplary machine learning model using a configurable nonlinear activation function circuit Figure 4 shows the dataflow 400 of an exemplary machine learning model that implements a configurable nonlinear activation function circuit as described above in Figures 1 to 3.

[0065] In flow 400, input data is stored in an input data buffer 401 (e.g., layer input data for a machine learning model) and then supplied to a multiply and accumulate (MAC) circuit 402. The MAC circuit 402 may generally be configured to perform vector, array, and matrix multiplication and accumulation operations, such as those frequently used in convolutional neural networks. In some examples, the MAC circuit 402 may include one or more compute-in-memory (CIM) arrays. Alternatively, or in addition to, the MAC circuit 402 may include a digital multiply and accumulate (DMAC). In further examples, the multiply and accumulate circuit 402 may be part of a machine learning accelerator such as a neural processing unit (NPU), or another type of processing unit optimized for performing machine learning operations. In other implementations, the MAC circuit 402 can be replaced by a vector / matrix or matrix / matrix processing engine.

[0066] The MAC circuit 402 processes the input data using weight data (e.g., neural network weight data) to generate pre-activation data. For example, the MAC circuit 402 may process input data to layers of a neural network model and generate pre-activation data as output.

[0067] The pre-activation data is fed into a configurable nonlinear activation (CNLA) function circuit 404, configured to generate output data (e.g., activation) based on a set nonlinear activation function. The output data can then be stored in an output data buffer 405 for subsequent use, such as for processing another layer in a machine learning model or as output from a machine learning model.

[0068] The CNLA function circuit 404 may be configured using the setting parameters described for the CNLA function circuit 100 in Figure 1 and / or the approximator 300 in Figure 3, as well as the setting parameters described in Tables 1 and 2. Furthermore, the CNLA function circuit 404 may be configured to access a lookup table depending on the set activation function.

[0069] In some cases, the setting parameters may include identifying a nonlinear activation function to be applied to the input data. Based on the determined nonlinear activation function, appropriate parameters (such as those in Table 2) can be retrieved from memory (e.g., registers) and applied to the CNLA function circuit 404, thereby setting up the CNLA function circuit to process the input data. In some examples, a finite state machine, a memory control unit, or another controller can perform the setting of the CNLA function circuit 404.

[0070] In particular, the CNLA circuit 404 may be configured to process multiple batches of input data using the same settings, or its settings may be updated for each new batch of input data. Thus, the CNLA circuit 404 provides a highly flexible and efficient means of performing configurable nonlinear activation for machine learning tasks such as training and inference.

[0071] An illustrative method for performing processing using a configurable nonlinear activation function circuit. Figure 5 shows an exemplary method 500 for performing the process using a configurable nonlinear activation function circuit.

[0072] Method 500 begins in step 502 with determining a nonlinear activation function to apply to the input data. For example, the nonlinear activation function may be one of the functions listed in Table 2, or another nonlinear activation function.

[0073] Next, method 500 proceeds to step 504, in which a set of parameters for a configurable nonlinear activation function circuit is determined based on the determined nonlinear activation function. For example, the parameters of the determined nonlinear activation function may be those shown in Tables 1 and 2 above.

[0074] Next, method 500 proceeds to step 506, in which the input data is processed by a nonlinear activation function circuit that can be configured based on a set of parameters to generate output data. For example, the output data may be activation data for layers of a neural network model.

[0075] In some examples, the parameter set includes a combination of one or more gain parameters, constant parameters, and one or more approximation functions for application to the input data via a configurable nonlinear activation function circuit. For example, the parameter set may be as described above with respect to Figures 1 and 2 and Table 1.

[0076] In some examples, method 500 further includes retrieving a set of parameters from memory based on a determined nonlinear activation function. In some examples, the memory may be one or more registers that store the parameter values.

[0077] In some examples, a configurable nonlinear activation function circuit includes a first approximator configured to approximate a first function from one or more approximation functions, a second approximator configured to approximate a second function from one or more approximation functions, a first gain multiplier configured to multiply the first gain value based on one or more gain parameters, and a constant adder configured to add a constant value, as illustrated and described with respect to Figure 1.

[0078] In some examples, a configurable nonlinear activation function circuit includes a first bypass configured to bypass a first approximator. In some examples, a configurable nonlinear activation function circuit includes a second bypass configured to bypass a second approximator. In some examples, a configurable nonlinear activation function circuit includes an input data bypass configured to bypass the first approximator and feed input data to the second approximator.

[0079] In some examples, at least one of the first and second approximators is a cubic approximator. In some examples, the other of the first and second approximators is either a quadratic or linear approximator. In some examples, the other of the first and second approximators is configured to perform a min / max function, as shown for path 306B in Figure 3. In some examples, the other of the first and second approximators is configured to access a lookup table for approximations, as shown for path 306C in Figure 3.

[0080] In some examples, both the first and second approximators are cubic approximators.

[0081] Figure 5 is just one example, and it should be noted that in other examples, methods such as those described herein may be implemented with more steps, fewer steps, and / or different steps.

[0082] Exemplary CNLA architecture for softmax operations using parallel input data Figure 6 shows an exemplary architecture 600 for performing a softmax operation using parallel input data with a CNLA function circuit.

[0083] The softmax (SM) function is used in a variety of machine learning models such as many neural network (NN) architectures. For example, the SM function is often used in the last layer of a neural network (after a fully connected (FC) / dense layer) to provide multi-category classification (such as digit recognition). SM is also used in attention-based computations (e.g., transformer models). Generally, SM maps the output of a neuron (e.g., a set of values within a tensor or vector) to an interval (e.g., values between 0 and 1) and ensures that the sum of the mapped values is 1.

[0084] As discussed above and discussed in more detail below, exponential, logarithmic, and natural logarithm (ln) functions can be computed using the CNLA architecture described above, such as by using a hardware look-up table. Thus, as described in more detail below, a CNLA circuit can be configured to provide an SM function (also referred to as an approximate SM function). In some aspects, a CNLA circuit is configured to provide an exact SM function if configured to perform exponential and logarithmic operations, and is configured to provide an approximate SM function if configured to perform approximate exponential and logarithmic operations (e.g., using a look-up table).

[0085] In at least one aspect, the SM function is

Number

Number

[0086] Furthermore, in some embodiments, log-softmax (Log(SM)) can be used in various architectures. In one embodiment, the log-softmax function is:

number

number

[0087] As will be described in more detail below, using the illustrated architecture 600, an approximate logarithmic softmax value 645 can be generated using a combination of CNLA blocks. In some embodiments, if the original (linear) softmax region is desired, the logarithmic softmax can be processed using another CNLA (e.g., CNLA circuit 635), as will be described in more detail below. This may be commonly referred to herein as a "two-step softmax operation".

[0088] Furthermore, in some embodiments, when the softmax function is used for multi-category classification, the simplified calculation can come quite close to the actual value by simply ignoring the denominator of the SM function defined above (for example, here

number

[0089] In the illustrated architecture 600, the computing array 605 provides input data (e.g., tensors or vectors) to be processed using softmax operations (e.g., approximate softmax and / or approximate logarithmic softmax operations). In the illustrated example, the computing array 605 is configured to output sets of data (e.g., multiple elements in a vector) in parallel. For example, the computing array 605 could be a CIM array. The computing array 605 can generally be used to perform any process or operation, such as generating the output of a layer of a neural network (e.g., by multiplying the input data by a set of weights associated with the layer).

[0090] In the illustrated example, two different processing paths are shown (for example, a first path including elements 615A, 620A, 625A, 640A, 645A, 650A, and 655A, and a second path including elements 615B, 620B, 625B, 640B, 645B, 650B, and 655B). In the illustrated example, architecture 600 further includes several common or shared elements across the processing paths, including max blocks 610, adder blocks 630, and CNLA circuits 635, which are described in more detail below.

[0091] Two paths are shown to clarify the concept, but any number of processing paths can exist within architecture 600 (as shown by ellipse 612). In at least one aspect, each path is associated with a corresponding output element of computing array 605 (e.g., a corresponding value in a vector or tensor). That is, each element x in vector x j It may have corresponding paths (including corresponding CNLA circuits 625). For example, if there are 64 elements in the vector, the architecture 600 may include 64 CNLA circuits 625. In at least one embodiment, each processing path corresponds to a channel of output generated by the computing array 605.

[0092] As shown in the figure, each output element of the computing array 605 (e.g., each value in vector x) is provided to the max block 610. The max block 610, which can be implemented using hardware or software, generally corresponds to a computing component that identifies and outputs the maximum value of its input data. Thus, in the illustrated embodiment, the max block 610 provides x (regardless of the number of values ​​in x or the number of processing paths included in architecture 600). max Identify x from x max The values ​​are provided to operations 615A and 615B. In one embodiment, the max block 610 can identify the maximum value by evaluating all input values ​​in parallel. Furthermore, each operation 615A and 615B receives a corresponding element from the computing array 605. That is, operation 615A receives the first value x a While it may receive the second value x, operation 615B is the second value x b It is possible to receive it.

[0093] In the illustrated example, operations 615A and 615B (collectively referred to as "operation 615") are subtraction operations, where x max is x j It is subtracted from. Specifically, operation 615A is x a -x max Calculate, while operation 615B is x b -x max The following is calculated. As shown in the figure, these values ​​are then provided to the corresponding multiplexers 620A and 620B (collectively, "multiplexer 620").

[0094] In some embodiments, the multiplexer 620 is used to enable the performance of a two-stage softmax operation (for example, when the desired output is a linear SM), as described above and in more detail below. In one embodiment, when an approximate logarithmic softmax is desired and / or when a single-stage softmax operation is used (or during the first stage of a two-stage operation), the multiplexer 620 may be used to provide the output of the operation 615 directly to the CNLA circuit 625. In the illustrated example, the outputs of multiplexers 615A and 615B are also provided to operations 640A and 640B, respectively, which are described in more detail below.

[0095] As shown in the figure, the output of the multiplexer 620 (or, as described above, the output of the arithmetic unit 615) is then provided to the respective CNLA circuits 625. Specifically, the output of the multiplexer 620A (for example, x a The output (calculated based on x) is provided to the first CNLA circuit 625A and the output of the multiplexer 620B (e.g., x b (calculated based on) is provided to the second CNLA circuit 625B.

[0096] In at least one embodiment, the CNLA circuits 625A and 625B (collectively, "CNLA circuits 625") may correspond to the CNLA function circuit 100 in Figure 1. In a particular illustrated architecture 600, the CNLA circuits 625 are configured to perform exponential operations or functions. That is, the CNLA circuits 625 take an input (e.g., e n To calculate or approximate the exponential output, the system is configured to calculate (or approximate) the exponential output based on the input data provided to the CNLA circuit 625, where n is the input data provided to the CNLA circuit 625.

[0097] In one embodiment, to perform this exponential operation, the CNLA circuit 625 may use a gain parameter including a dependent parameter value of 0, an independent parameter value of 1, and a constant value of 0, with the first function being bypassed and the second function being an exponential lookup table. In some embodiments, the first function may be an exponential lookup table, but the second function is bypassed. In this way, the CNLA circuit 625 can provide an exponential function.

[0098] Therefore, as shown in the figure, the output of each CNLA circuit 625 is

number

number

number

number

[0099] In at least one embodiment, the CNLA circuit 635 may correspond to the CNLA function circuit 100 in Figure 1. In a particular illustrated architecture 600, the CNLA circuit 635 is configured to perform a logarithmic operation or function. That is, the CNLA circuit 635 is configured to compute (or approximate) a logarithmic (e.g., natural logarithmic) output based on an input (e.g., to compute or approximate ln(n), where n is the input data provided to the CNLA circuit 635).

[0100] In one embodiment, to perform this exponential operation, the CNLA circuit 635 may use a gain parameter including a dependent parameter value of 0, an independent parameter value of 1, and a constant value of 0, where the first function is bypassed and the second function is a logarithmic lookup table (e.g., a natural logarithmic lookup table). In some embodiments, the first function may be a logarithmic lookup table, but the second function is bypassed. In this way, the CNLA circuit 635 can provide a logarithmic function.

[0101] Therefore, as shown in the diagram, the output of the CNLA circuit 635 is

number

[0102] In other words, operation 640 is

number

number

number

[0103] As described above and shown in the example in the figure, the output of operation 640 is therefore the logarithmic softmax values ​​645A and 645B (collectively referred to as "logarithmic softmax value 645"). That is, the logarithmic softmax value 645A is log(SM(x a )) may be equal to or approximate to log(SM(x), and the logarithmic softmax value 645B is log(SM(x b )) may be equal to or approximate ). In some embodiments, if the log softmax is the desired output, the log softmax value 645 may be provided as an output from the architecture 600 (e.g., as an output from the model or as an input to subsequent layers of the neural network).

[0104] In at least one embodiment, as described above, in a single-step operation, the system may directly provide the output of the adder block 630 to the operation 640, rather than using the CNLA circuit 635. This can, in some implementations, enable more efficient generation of approximate softmax values.

[0105] As described above, if the desired output of the architecture is a linear softmax value and the architecture uses two-stage arithmetic, then any optional paths 650A and 650B (collectively, “path 650”) may be used. In the illustrated example, these paths 650 return the generated logarithmic softmax value 645 to the multiplexer 620 during a second or subsequent cycle or stage. That is, the illustrated components may process the output of the computing array 605 to generate the logarithmic softmax value 645 in a first stage (e.g., during a first set of one or more clock cycles). During a subsequent stage (e.g., during a second set of one or more clock cycles), the logarithmic softmax value 645 can be returned to the multiplexer 620 / CNLA circuit 625.

[0106] In some embodiments, during this second stage, the multiplexer 620 can pass the logarithmic softmax values ​​645 to the corresponding CNLA circuits 625. That is, each multiplexer 620 can directly provide the generated logarithmic softmax values ​​645 to the corresponding CNLA circuits 625. Specifically, the logarithmic softmax value 645A is provided as an input to CNLA circuit 625A, and the logarithmic softmax value 645B is provided as an input to CNLA circuit 625B. The illustrated example shows providing the logarithmic softmax values ​​645 back to the CNLA circuits 625, but in some embodiments, the architecture may provide further exponential operations using a second set of CNLA circuits separate from the CNLA circuits 625.

[0107] As illustrated and explained above, the CNLA circuit 625 is configured to calculate an exponential output based on the input data. Thus, upon receiving a logarithmic softmax value 645, each CNLA circuit 625 generates and outputs the corresponding linear softmax value 655 (e.g., linear softmax values ​​655A and 655B). That is, each CNLA circuit 625 calculates exp(log(SM(x)) for the corresponding jth value from the input vector. j))) can be calculated. In this way, the output of the CNLA circuit 625 is equal to or approximates the SM function described above.

[0108] In one embodiment, these linear softmax values ​​655 may then be provided as an output from the architecture 600 (for example, as an output from the model or as an input to subsequent layers of the neural network).

[0109] Exemplary CNLA architecture for softmax arithmetic using sequential input data Figure 7 shows an exemplary architecture 700 of a CNLA function circuit for performing a softmax operation using sequential input data.

[0110] As will be described in more detail below, using the illustrated architecture 700, an approximate logarithmic softmax value 745 can be generated using a combination of CNLA blocks. In some embodiments, if the original (linear) softmax region is desired, the logarithmic softmax can be processed using another CNLA (e.g., CNLA circuit 735), as described above and in more detail below. This may be commonly referred to herein as a "two-step softmax operation".

[0111] Furthermore, as mentioned above, when the softmax function is used for multi-category classification, the simplified calculation can come quite close to the actual value by simply ignoring the denominator of the SM function defined above (for example, here

number

[0112] In the illustrated architecture 700, the sequential circuit 705 provides input data (e.g., tensor or vector values ​​or data elements) that are processed using softmax operations (e.g., approximate softmax and / or approximate logarithmic softmax operations). In the illustrated example, the sequential circuit 705 is configured to output data elements sequentially. That is, while the architecture 600 in Figure 6 is configured for parallel data input, the illustrated architecture 700 is configured to process sequential input (e.g., data elements are output one at a time by the sequential circuit 705). For example, the sequential circuit 705 could be a DMAC circuit. The sequential circuit 705 can generally be used to perform any process or operation, such as generating the output of a layer in a neural network (e.g., by sequentially multiplying the input data elements by the corresponding weights associated with the layer).

[0113] In the illustrated example, in contrast to the architecture 600 shown in Figure 6, a single processing path is used to process the output of the sequential circuit 705, and architecture 700 processes the output sequentially. That is, using the illustrated processing path, each element x in the vector x is processed sequentially. j These can be processed sequentially. For example, using the illustrated architecture 700, to start processing, each clock cycle, a new element x j This can be output by the sequential circuit 705. In at least one embodiment, architecture 700 can be used to sequentially process data elements in a given channel of data output by layers of a neural network, and a separate architecture 700 can be used to process each respective channel.

[0114] As shown in the figure, each output element is first provided to buffer 707. In one embodiment, buffer 707 is a memory or storage component (e.g., a register file) that buffers or stores each output element from sequential circuit 705. For example, sequential circuit 705 continues until all vectors x have been generated. j By outputting, buffer 707 can store each element until the entire vector is stored. As shown in the figure, buffer 707 outputs the elements to max block 710. In some embodiments, buffer 707 is shown, but the output of sequential circuit 705 may instead be provided directly to max block 710. That is, buffer 707 may be used to provide input to arithmetic 715, and max block 710 may receive the buffer output or may receive input directly (sequentially) from sequential circuit 705.

[0115] The max block 727, which can be implemented using hardware or software, generally corresponds to a computing component that identifies and outputs the maximum value of its input data. Thus, in the illustrated embodiment, the max block 710 is x to x max Identify. In some embodiments, using buffer 707, the max block 710 can evaluate the entire vector x at once. In some embodiments where buffer 707 is not used, the max block 710 evaluates each x j As each value x is received, it can be evaluated sequentially. For example, the max block 707 evaluates each newly received value x j Evaluate this value and determine the x currently stored by max block 707 max It is possible to determine if it is greater than or equal to the new / current x. If so, the new value is the new / current x max It can be buffered as such. Once all data elements have been evaluated, the max block 707 can output the determined maximum value.

[0116] In the illustrated example, the max block 710 and buffer 707 then output data to the operation 715. That is, the max block 710 outputs the maximum value of x to the operation 715, and the operation 715 also receives the output of buffer 707 (for example, the entire vector x). In the illustrated example, the operation 715 is a subtraction operation, and x max (Output from max block 707) is subtracted from x (Output from buffer 707). Specifically, operation 715 subtracts xx for all values ​​in x. max The values ​​are calculated. As shown in the figure, these values ​​are then provided to the multiplexer 720. In the illustrated example, the output of the operation 715 is also provided to the buffer 737.

[0117] In one embodiment, similar to buffer 707, buffer 737 is a memory or storage component (e.g., a register file) that buffers or stores each output element from operation 715. For example, operation 715 continues until all of vector x is generated / evaluated. j -x max When outputting, buffer 737 will remain until the entire vector is processed (for example, each value x j -x max Each element can be stored (until the calculation is performed).

[0118] In the illustrated example, the multiplexer 720 is used to enable a two-stage softmax operation (for example, when the desired output is a linear SM) as described above and in more detail below. In one embodiment, when an approximate logarithmic softmax is desired and / or when a single-stage softmax operation is used (or during the first stage of a two-stage operation), the multiplexer 720 may be used to provide the output of the operation 715 directly to the CNLA circuit 725.

[0119] In at least one embodiment, the CNLA circuit 725 may correspond to the CNLA function circuit 100 in Figure 1. In a particular illustrated architecture 700, the CNLA circuit 725 is configured to perform an exponential operation or a function. That is, the CNLA circuit 725 takes an input (e.g., en To calculate or approximate the exponential output, the system is configured to calculate (or approximate) the exponential output based on the input data provided to the CNLA circuit 725, where n is the input data provided to the CNLA circuit 725.

[0120] In one embodiment, to perform this exponential operation, the CNLA circuit 725 may use a gain parameter including a dependent parameter value of 0, an independent parameter value of 1, and a constant value of 0, with the first function being bypassed and the second function being an exponential lookup table. In some embodiments, the first function may be an exponential lookup table, but the second function is bypassed. In this way, the CNLA circuit 725 can provide an exponential function.

[0121] Therefore, as shown in the diagram, the output of the CNLA circuit 725 is

number

[0122] An adder block 730, which can be implemented using hardware or software, generally corresponds to a computing component that sums up received input values ​​and outputs the sum. In one embodiment, the adder block 730 can similarly sum up received input values ​​sequentially as these values ​​are received. That is, the adder block 730 sums each newly received value

number

[0123] In the illustrated example, the generated sum is output to the CNLA circuit 735 by the addition block 730. In at least one embodiment, the CNLA circuit 735 may correspond to the CNLA function circuit 100 in Figure 1. In a particular illustrated architecture 700, the CNLA circuit 735 is configured to perform a logarithmic operation or function. That is, the CNLA circuit 735 is configured to calculate (or approximate) a logarithmic (e.g., natural logarithmic) output based on an input (e.g., to calculate or approximate ln(n), where n is the input data provided to the CNLA circuit 735).

[0124] In one embodiment, to perform this exponential operation, the CNLA circuit 735 may use a gain parameter including a dependent parameter value of 0, an independent parameter value of 1, and a constant value of 0, where the first function is bypassed and the second function is a logarithmic lookup table (e.g., a natural logarithmic lookup table). In some embodiments, the first function may be a logarithmic lookup table, but the second function is bypassed. In this way, the CNLA circuit 735 can provide a logarithmic function.

[0125] Therefore, as shown in the diagram, the output of the CNLA circuit 735 is

number

number

[0126] As described above and as shown in the illustrated example, the output of the operation 740 is the log softmax value 745 for all x values ​​output by the sequential circuit 705. In some embodiments, if the log softmax is the desired output, the log softmax value 745 may be provided as an output from the architecture 700 (for example, as an output from the model or as an input to a subsequent layer of the neural network).

[0127] In at least one embodiment, as described above, in a single-stage operation, the system may bypass the CNLA circuit 735 and directly provide the output of the adder block 730 to the operation 740. This can enable more efficient generation of approximate softmax values ​​in some implementations.

[0128] As described above, if the desired output of the architecture is a linear softmax value and the architecture uses a two-stage operation, an optional path 750 may be used. In the illustrated example, this path 750 returns the generated logarithmic softmax value 745 to the multiplexer 720 during a second or subsequent cycle or stage. That is, the illustrated components may process the output of the sequential circuit 705 to generate the logarithmic softmax value 745 in a first stage (e.g., during a first set of one or more clock cycles). During a subsequent stage (e.g., during a second set of one or more clock cycles), the logarithmic softmax value 745 can be returned to the multiplexer 720 / CNLA circuit 725.

[0129] The illustrated example shows that the logarithmic softmax value 745 is provided to be returned to the CNLA circuit 725 (via the multiplexer 720), but in some embodiments, the architecture may provide further exponential calculations using a second CNLA circuit separate from the CNLA circuit 725.

[0130] As illustrated and explained above, the CNLA circuit 725 is configured to calculate an exponential output based on the input data. Therefore, upon receiving a logarithmic softmax value 745, the CNLA circuit 725 generates and outputs a corresponding set of linear softmax values ​​755. That is, the CNLA circuit 725 can calculate exp(x-log(SM)) for all x values ​​in the input vector. In this way, the output of the CNLA circuit 725 is equal to or approximates the SM function described above.

[0131] In one embodiment, these linear softmax values ​​755 may then be provided as an output from the architecture 700 (for example, as an output from the model or as an input to subsequent layers of the neural network).

[0132] An exemplary method for performing an approximate softmax calculation using a CNLA function circuit. Figure 8 shows an exemplary method 800 for performing an approximate softmax operation using a configurable nonlinear activation function circuit.

[0133] In block 802, an exponential output is generated by processing the input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function (e.g., the first approximator 102 in Figure 1, the approximator block 206A in Figure 2, the approximator 300 in Figure 3, the CNLA circuit 625 in Figure 6, and / or the CNLA circuit 725 in Figure 7).

[0134] In block 804, the exponential outputs of one or more first configurable nonlinear activation function circuits are summed (for example, using the summing block 630 in Figure 6 and / or the summing block 730 in Figure 7).

[0135] In block 806, the approximate logarithmic softmax output is generated by processing the sum exponential output using a second configurable nonlinear activation function circuit configured to perform a natural logarithmic function (e.g., the second approximator 104 in Figure 1, approximator block 206B in Figure 2, approximator 300 in Figure 3, CNLA circuit 635 in Figure 6, and / or CNLA circuit 735 in Figure 7).

[0136] In some embodiments, Method 800 further includes generating an approximate softmax of the input data by processing the approximate logarithmic softmax of the input data using one or more first configurable nonlinear activation function circuits.

[0137] In some embodiments, one or more first configurable nonlinear activation function circuits comprise a plurality of first configurable nonlinear activation circuits, each associated with a corresponding output element from a parallel computing array.

[0138] In some embodiments, the method 800 further includes determining a maximum value from the output elements of a parallelized computing array and providing the maximum value from the output elements to one or more first configurable nonlinear activation function circuits.

[0139] In some embodiments, one or more first configurable nonlinear activation function circuits comprise a single first configurable nonlinear activation circuit that receives input data from a sequential computing circuit.

[0140] In some embodiments, method 800 further includes determining the maximum output from a single first configurable nonlinear activation circuit.

[0141] In some embodiments, method 800 further includes buffering the output from a sequential computing circuit.

[0142] In some embodiments, the method 800 further includes determining a nonlinear activation function to apply to input data, determining a set of parameters for a configurable nonlinear activation function circuit based on the determined nonlinear activation function, and processing the input data using the configurable nonlinear activation function circuit based on the set of parameters to generate output data.

[0143] In some embodiments, method 800 further includes retrieving a set of parameters from memory based on a determined nonlinear activation function.

[0144] In some embodiments, the set of parameters includes a combination of one or more gain parameters, constant parameters, and one or more approximation functions for application to input data via a configurable nonlinear activation function circuit.

[0145] In some embodiments, at least one of the first configurable nonlinear activation function circuits includes a first approximator configured to approximate a first function from one or more approximation functions, a second approximator configured to approximate a second function from one or more approximation functions, a first gain multiplier configured to multiply a first gain value based on one or more gain parameters, and a constant adder configured to add a constant value based on a constant parameter.

[0146] In some embodiments, at least one of the first configurable nonlinear activation function circuits further comprises a first bypass configured to bypass a first approximator, a second bypass configured to bypass a second approximator, and an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0147] In some embodiments, the determined nonlinear activation function includes an exponential function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is an exponential lookup table.

[0148] Exemplary Processing System Figure 9 shows an exemplary processing system 900 that may be configured to implement the systems, technologies, architectures, and methods described herein, including with respect to Figures 1 to 8.

[0149] The processing system 900 includes a central processing unit (CPU) 902, which in some examples may be a multi-core CPU. Instructions executed by the CPU 902 may be loaded, for example, from program memory associated with the CPU 902, or from a memory partition 924.

[0150] The processing system 900 also includes additional processing components organized for specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, a multimedia processing unit 910, and a wireless connectivity component 912.

[0151] NPUs, such as the 908, are generally specialized circuits configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, and more. NPUs are sometimes alternatively called neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), or vision processing units (VPUs).

[0152] NPUs, such as the 908, can be configured to accelerate performance on common machine learning tasks, including image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in others, they may be part of a dedicated machine learning accelerator device.

[0153] NPUs can be optimized for training or inference, or in some cases, configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0154] NPUs, designed to accelerate training, are generally configured to accelerate the optimization of new models, a highly computationally intensive operation that involves inputting existing datasets (often labeled or tagged), iterating through those datasets, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients that reduce prediction errors.

[0155] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs may be configured to take new data as input and process the data rapidly through a model that has already been trained to produce model outputs (e.g., inferences).

[0156] In some embodiments, the NPU908 may be implemented as part of one or more of the CPU902, GPU904, and / or DSP906.

[0157] In some embodiments, the wireless connectivity component 912 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 912 is further connected to one or more antennas 914.

[0158] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920 which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0159] The processing system 900 may also include one or more input and / or output devices 922, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0160] In some examples, one or more of the processors in the processing system 900 may be based on an ARM or RISC-V instruction set.

[0161] The processing system 900 also includes various circuits in various embodiments described herein.

[0162] In this example, the processing system 900 includes a compute-in-memory (CIM) circuit 926, which may be configured to perform an efficient multiplication-accumulation (MAC) function for processing machine learning model data. The processing system 900 further includes a configurable nonlinear activation (CNLA) function circuit 928. In some cases, the CNLA function circuit 928 may be similar to the CNLA function circuit 200 described with respect to Figure 2. The CNLA function circuit 928, as well as other circuits not shown, may be configured to perform various embodiments of the methods described herein, such as flow 400 with respect to Figure 4.

[0163] In some examples, the CNLA function circuit 928 may be implemented as part of another processing unit such as the CPU 902, GPU 904, DSP 906, or NPU 908.

[0164] The processing system 900 also includes memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, memory 924 includes computer executable components that can be executed by one or more of the above components of the processing system 900.

[0165] In particular, in this example, memory 924 includes a decision component 924A, a setting component 924B, a processing component 924C, a retrieval component 924D, a nonlinear activation function parameter 924E, a lookup table (one or more) 924F, and model parameters 924G (e.g., weights, biases, and other machine learning model parameters). One or more of the illustrated components, as well as other components not illustrated, may be configured to implement various aspects of the methods described herein.

[0166] In general, the processing system 900 and / or its components may be configured to perform the methods described herein.

[0167] In particular, in other embodiments, the embodiments of the processing system 900 may be omitted, for example, when the processing system 900 is a server computer. For example, the multimedia components 910, wireless connectivity 912, sensors 916, ISP 918, and / or navigation components 920 may be omitted in other embodiments. Furthermore, the embodiments of the processing system 900 may be distributed.

[0168] Please note that Figure 9 is merely one example, and other examples may use alternative processing systems with more, fewer, and / or different components.

[0169] Exemplary clause Implementation examples are described in the following numbered clauses.

[0170] Clause 1: A processor comprising: one or more first configurable nonlinear activation function circuits configured to perform an exponential function on input data; an adder circuit configured to receive output data from one or more first configurable nonlinear activation function circuits; and a second configurable nonlinear activation function circuit configured to receive output data from the adder circuit, perform a natural logarithmic function, and output an approximate logarithmic softmax of the input data.

[0171] Clause 2: The processor according to Clause 1, wherein a second configurable nonlinear activation function circuit is configured to output an approximate logarithmic softmax of the input data during a first cycle, and during a second cycle following the first cycle, the approximate logarithmic softmax of the input data is provided as an input to one or more first configurable nonlinear activation function circuits, and one or more first configurable nonlinear activation function circuits are configured to output an approximate softmax of the input data based on the approximate logarithmic softmax of the input data.

[0172] Clause 3: The processor according to Clause 1 or 2, comprising one or more first configurable nonlinear activation function circuits, each associated with a corresponding output element from a parallel computing array.

[0173] Clause 4: The processor according to any one of Clauses 1 to 3, further comprising a max circuit, the max circuit configured to receive output elements from a parallel computing array as input and output a maximum value from the output elements to one or more first configurable nonlinear activation function circuits.

[0174] Clause 5: The processor according to any one of Clauses 1 to 4, comprising one or more first configurable nonlinear activation function circuits, each comprising a single first configurable nonlinear activation circuit configured to receive input data from a sequential computing circuit.

[0175] Clause 6: The processor according to any one of Clauses 1 to 5, further comprising a max circuit, the max circuit configured to take the output from a single first configurable nonlinear activation circuit as input and output a maximum value from a sequential computing circuit.

[0176] Clause 7: The processor described in any one of Clauses 1 to 6, further comprising a memory buffer configured to buffer the output from a sequential computing circuit.

[0177] Clause 8: The processor according to any one of Clauses 1 to 7, wherein at least one of the first configurable nonlinear activation function circuits is configured to determine a nonlinear activation function for application to input data, determine a set of parameters for the nonlinear activation function based on the determined nonlinear activation function, and generate output data based on the application of the set of parameters for the nonlinear activation function.

[0178] Clause 9: The processor according to any one of Clauses 1 to 8, wherein at least one of one of a set of first configurable nonlinear activation function circuits comprises: a first approximator configured to approximate a first function using one or more first function parameters from a set of parameters; a second approximator configured to approximate a second function using one or more second function parameters from a set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters from a set of parameters; and a constant adder configured to add a constant value based on a constant parameter from a set of parameters.

[0179] Clause 10: A processor according to any one of Clauses 1 through 9, wherein both the first and second approximators are cubic approximators.

[0180] Clause 11: A processor according to any one of Clauses 1 to 10, wherein either the first or second approximator is a cubic approximator.

[0181] Clause 12: A processor according to any one of Clauses 1 to 11, wherein the first approximator or the other of the second approximators is either a quadratic approximator or a linear approximator.

[0182] Clause 13: A processor as described in any one of Clauses 1 through 12, wherein the other of the first or second approximator is configured to obtain an approximation and access a lookup table.

[0183] Clause 14: A processor as described in any one of Clauses 1 to 13, wherein the first approximator or the other of the second approximator is configured to perform a minimum or maximum function.

[0184] Clause 15: The processor described in any one of Clauses 1 to 14, wherein the determined nonlinear activation function includes an exponential function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is an exponential lookup table.

[0185] Clause 16: The processor according to any one of Clauses 1 to 15, comprising: a second configurable nonlinear activation function circuit, comprising: a first approximator configured to approximate a first function using one or more first function parameters from a set of parameters; a second approximator configured to approximate a second function using one or more second function parameters from a set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters from a set of parameters; and a constant adder configured to add a constant value based on a constant parameter from a set of parameters, wherein the gain parameters include dependent parameter values ​​of 0 and independent parameter values ​​of 1, the constant value is 0, the first function is bypassed, and the second function is a natural logarithmic lookup table.

[0186] Clause 17: A method for processing input data by a set of configurable nonlinear activation function circuits, comprising: generating an exponential output by processing the input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function; adding the exponential outputs of one or more first configurable nonlinear activation function circuits; and generating an approximate logarithmic softmax output by processing the added exponential outputs using a second configurable nonlinear activation function circuit configured to perform a natural logarithmic function.

[0187] Clause 18: The method according to Clause 17, further comprising generating an approximate softmax of input data by processing the approximate log softmax of the input data using one or more first configurable nonlinear activation function circuits.

[0188] Clause 19: The method according to any one of Clauses 17 to 18, wherein one or more first configurable nonlinear activation function circuits comprises multiple first configurable nonlinear activation circuits, each associated with a corresponding output element from a parallelized computing array.

[0189] Clause 20: The method according to any one of Clauses 17 to 19, further comprising determining a maximum value from the output elements of a parallelized computing array and providing the maximum value from the output elements to one or more first configurable nonlinear activation function circuits.

[0190] Clause 21: The method according to any one of Clauses 17 to 20, wherein one or more first configurable nonlinear activation function circuits comprises a single first configurable nonlinear activation circuit that receives input data from a sequential computing circuit.

[0191] Clause 22: The method according to any one of Clauses 17 to 21, further comprising determining the maximum output from a single first configurable nonlinear activation circuit.

[0192] Clause 23: The method according to any one of Clauses 17 to 22, further comprising buffering the output from a sequential computing circuit.

[0193] Clause 24: The method according to any one of Clauses 17 to 23, further comprising determining a nonlinear activation function for application to input data, determining a set of parameters for a configurable nonlinear activation function circuit based on the determined nonlinear activation function, and processing the input data using the configurable nonlinear activation function circuit based on the set of parameters to generate output data.

[0194] Clause 25: The method according to any one of Clauses 17 to 24, further comprising retrieving a set of parameters from memory based on a determined nonlinear activation function.

[0195] Clause 26: The method described in any one of Clauses 17 to 25, wherein the set of parameters includes a combination of one or more gain parameters, constant parameters, and one or more approximation functions for application to input data via a configurable nonlinear activation function circuit.

[0196] Clause 27: The method according to any one of Clauses 17 to 26, wherein at least one of the first configurable nonlinear activation function circuits comprises: a first approximator configured to approximate a first function from one or more approximation functions; a second approximator configured to approximate a second function from one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value based on a constant parameter.

[0197] Clause 28: The method according to any one of Clauses 17 to 27, wherein at least one of the first configurable nonlinear activation function circuits further comprises a first bypass configured to bypass a first approximator, a second bypass configured to bypass a second approximator, and an input data bypass configured to bypass the first approximator and provide input data to the second approximator.

[0198] Clause 29: The method according to any one of Clauses 17 to 28, wherein the determined nonlinear activation function includes an exponential function, the gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is an exponential lookup table.

[0199] Clause 30: The method according to any one of Clauses 17 to 29, wherein a second configurable nonlinear activation function circuit comprises: a first approximator configured to approximate a first function using one or more first function parameters from a set of parameters; a second approximator configured to approximate a second function using one or more second function parameters from a set of parameters; a gain multiplier configured to multiply a gain value based on one or more gain parameters from a set of parameters; and a constant adder configured to add a constant value based on a constant parameter from a set of parameters, wherein the gain parameters include a dependent parameter value of 0 and an independent parameter value of 1, the constant value is 0, the first function is bypassed, and the second function is a natural logarithmic lookup table.

[0200] Clause 31: A processing system comprising memory containing computer executable instructions and one or more processors configured to execute computer executable instructions and cause the processing system to carry out the methods described in any one of Clauses 17 to 30.

[0201] Clause 32: A processing system comprising means for carrying out the method described in any one of Clauses 17 to 30.

[0202] Clause 33: A non-temporary computer-readable medium containing computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to carry out the method described in any one of Clauses 17 to 30.

[0203] Clause 34: A computer program product embodied on a computer-readable storage medium, comprising code that implements the method described in any one of Clauses 17 to 30.

[0204] Additional considerations The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may apply to other embodiments. For example, changes may be made to the function and arrangement of the elements described without departing from the scope of this disclosure. Various examples may omit, replace, or add various procedures or components as needed. For example, the methods described may be carried out in an order different from the order in which they are described, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined with some other embodiments. For example, an apparatus can be implemented or a method can be practiced using any number of embodiments described herein. In addition, the scope of this disclosure is intended to encompass apparatus or methods that are practiced using other structures, functions, or structures and functions in addition to, or other than, the various embodiments of this disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0205] As used herein, the term “exemplary” means “serving as an example, case, or illustration.” No embodiment described herein as “exemplary” should be construed as necessarily preferable or advantageous to any other embodiment.

[0206] Where used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other sequence of a, b, and c).

[0207] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0208] As used herein, the term “connected to” may generally have meaning in data communication between corresponding elements that are connected to each other, in the context of sharing electronic signals and data between elements described herein. In some cases, elements may be directly connected to each other, for example, via one or more conductive traces, lines, or other conductive carriers that can carry signals and / or data between corresponding elements that are directly connected to each other. In other cases, elements may be indirectly connected to each other, for example, via one or more data buses or similar shared circuits and / or integrated circuit elements for communicating signals and data between corresponding elements that are indirectly connected to each other.

[0209] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of those methods can be replaced with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, can be modified without departing from the claims. Furthermore, various operations of the methods described above can be carried out by any preferred means capable of performing the corresponding function. These means may include, but are not limited to, various hardware and / or software components, and / or various hardware and / or software modules, including, but not limited to, circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations are shown in the figures, those operations may have corresponding equivalent means-plus-function components with similar numbering.

[0210] The following claims are not intended to be limited to the embodiments shown herein, but rather the full scope consistent with the language of the claims should be recognized. Within the claims, a singular reference to an element is intended to mean "one or more" rather than "one and only one" unless explicitly stated otherwise. Unless otherwise explicitly stated, the term "several" refers to one or more. The elements of the claims should not be construed under Section 112(f) of the U.S. Patent Act unless the element is expressly described using the phrase "means of" or, in the case of a method claim, the element is described using the phrase "steps of". All structural and functional equivalents of the elements in various embodiments described throughout this disclosure, whether known to those skilled in the art or to become known thereafter, are expressly incorporated by reference herein and intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether such disclosure is expressly stated in the claims or not.

Claims

1. It is a processor, One or more first configurable nonlinear activation function circuits configured to perform an exponential function on input data, An adder circuit configured to receive output data from one or more first configurable nonlinear activation function circuits, A second configurable nonlinear activation function circuit is configured to receive the output data of the summing circuit, execute a natural logarithm function, and output an approximate logarithmic softmax of the input data, A processor equipped with the following features.

2. The processor according to claim 1, wherein the second configurable nonlinear activation function circuit is configured to output the approximate logarithmic softmax of the input data during a first cycle, and during a second cycle following the first cycle, the approximate logarithmic softmax of the input data is provided as an input to the one or more first configurable nonlinear activation function circuits, and the one or more first configurable nonlinear activation function circuits are configured to output the approximate softmax of the input data based on the approximate logarithmic softmax of the input data.

3. The processor according to claim 1, comprising a plurality of first configurable nonlinear activation function circuits, each of which is associated with a corresponding output element from a parallel computing array.

4. It further includes a max circuit, and the max circuit is The output elements are received as input from the parallel computing array. The processor according to claim 3, configured to output a maximum value from the output element to one or more first configurable nonlinear activation function circuits.

5. The processor according to claim 1, wherein the one or more first configurable nonlinear activation function circuits include a single first configurable nonlinear activation circuit configured to receive the input data from a sequential computing circuit.

6. It further includes a max circuit, and the max circuit is The output from the single first configurable nonlinear activation circuit is received as input, The processor according to claim 5, configured to output a maximum value from the sequential computing circuit.

7. The processor according to claim 5, further comprising a memory buffer configured to buffer the output from the sequential computing circuit.

8. At least one of the first configurable nonlinear activation function circuits is A nonlinear activation function is determined for application to the aforementioned input data. Based on the determined nonlinear activation function, a set of parameters for the nonlinear activation function is determined. The processor according to claim 1, configured to generate output data based on the application of the set of parameters for the nonlinear activation function.

9. At least one of the one or more first configurable nonlinear activation function circuits is A first approximator configured to approximate a first function using one or more first function parameters from the set of parameters, A second approximator configured to approximate a second function using one or more second function parameters from the set of parameters, A gain multiplier configured to multiply the gain value based on one or more gain parameters from the set of parameters, A constant adder configured to add constant values ​​based on the constant parameters of the set of parameters, The processor according to claim 8, comprising:

10. The processor according to claim 9, wherein both the first approximator and the second approximator are cubic approximators.

11. The processor according to claim 9, wherein one of the first or second approximator is a cubic approximator.

12. The processor according to claim 11, wherein the other of the first or second approximator is a quadratic approximator or a linear approximator.

13. The processor according to claim 11, wherein the other of the first or second approximator is configured to obtain an approximate value and access a lookup table.

14. The processor according to claim 11, wherein the other of the first or second approximator is configured to perform a minimum or maximum function.

15. The determined nonlinear activation function includes an exponential function, The aforementioned gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1. The aforementioned constant value is 0. The first function is bypassed, The processor according to claim 9, wherein the second function is an exponential lookup table.

16. The second configurable nonlinear activation function circuit is, A first approximator configured to approximate a first function using one or more first function parameters from a set of parameters, A second approximator configured to approximate a second function using one or more second function parameters from the set of parameters, A gain multiplier configured to multiply the gain value based on one or more gain parameters from the set of parameters, A constant adder configured to add constant values ​​based on the constant parameters of the set of parameters, The aforementioned gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1. The aforementioned constant value is 0. The first function is bypassed, The processor according to claim 1, wherein the second function is a natural logarithmic lookup table.

17. A method for processing input data using a set of configurable nonlinear activation function circuits, The process involves generating an exponential output by processing input data using one or more first configurable nonlinear activation function circuits configured to perform an exponential function, Adding the exponential outputs of the one or more first configurable nonlinear activation function circuits, An approximate logarithmic softmax output is generated by processing the summed exponential output using a second configurable nonlinear activation function circuit configured to perform a natural logarithmic function, Methods that include...

18. The method according to claim 17, further comprising generating an approximate softmax of the input data by processing the approximate logarithmic softmax of the input data using the one or more first configurable nonlinear activation function circuits.

19. The method according to claim 17, wherein the one or more first configurable nonlinear activation function circuits comprise a plurality of first configurable nonlinear activation circuits, each associated with a corresponding output element from a parallelized computing array.

20. Determining the maximum value from the output elements of the parallel computing array, The method according to claim 19, further comprising providing the maximum value from the output element to the one or more first configurable nonlinear activation function circuits.

21. The method according to claim 17, wherein the one or more first configurable nonlinear activation function circuits include a single first configurable nonlinear activation circuit that receives input data from a sequential computing circuit.

22. The method according to claim 21, further comprising determining the maximum output from the single first configurable nonlinear activation circuit.

23. The method according to claim 21, further comprising buffering the output from the sequential computing circuit.

24. Determining a nonlinear activation function for application to input data, Based on the determined nonlinear activation function, a set of parameters for a configurable nonlinear activation function circuit is determined, The method according to claim 17, further comprising processing input data using the configurable nonlinear activation function circuit based on the set of parameters in order to generate output data.

25. The method according to claim 24, further comprising retrieving the set of parameters from memory based on the determined nonlinear activation function.

26. The method according to claim 24, wherein the set of parameters includes a combination of one or more gain parameters, constant parameters, and one or more approximation functions for application to the input data via the configurable nonlinear activation function circuit.

27. At least one of the first configurable nonlinear activation function circuits is A first approximator configured to approximate a first function among the one or more approximation functions, A second approximator configured to approximate a second function among one or more approximation functions, A first gain multiplier configured to multiply a first gain value based on one or more of the aforementioned gain parameters, The method according to claim 26, comprising a constant adder configured to add a constant value based on the constant parameter.

28. At least one of the first configurable nonlinear activation function circuits is A first bypass configured to bypass the first approximator, A second bypass configured to bypass the second approximator, The method according to claim 27, further comprising: an input data bypass configured to bypass the first approximator and provide the input data to the second approximator.

29. The determined nonlinear activation function includes an exponential function, The aforementioned gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1. The aforementioned constant value is 0. The first function is bypassed, The method according to claim 28, wherein the second function is an exponential lookup table.

30. The second configurable nonlinear activation function circuit is, A first approximator configured to approximate a first function using one or more first function parameters from a set of parameters, A second approximator configured to approximate a second function using one or more second function parameters from the set of parameters, A gain multiplier configured to multiply the gain value based on one or more gain parameters from the set of parameters, A constant adder configured to add constant values ​​based on the constant parameters of the set of parameters, The aforementioned gain parameter includes a dependent parameter value of 0 and an independent parameter value of 1. The aforementioned constant value is 0. The first function is bypassed, The method according to claim 17, wherein the second function is a natural logarithmic lookup table.

Images