Runtime linear approximation activation function implementation circuit and method based on curvature optimization

Abstract

This invention relates to the field of neural network hardware implementation technology, providing a runtime non-uniform piecewise linear approximation activation function implementation circuit and method based on curvature optimization. It utilizes the radius of curvature to derive the upper bound of the linear approximation error, and automatically selects non-uniform breakpoints using a greedy iterative selection algorithm. The slope and intercept of each segment are then solved using the least squares method and the BFGS global optimization algorithm. Hardware implementation is achieved by constructing a runtime linear approximation activation function implementation circuit consisting of a parameter memory, a parallel comparator array, a thermometer code encoder, and multiply-accumulate units. During circuit operation, the segment threshold, slope, and intercept parameters of the target activation function are written into the parameter memory, enabling dynamic updates of the activation function without interrupting the inference data stream. This decouples the mathematical expression of the function from the hardware topology, constructs a curvature-driven automated parameter generation process, and forms a dynamically configurable hardware architecture.

Description

Technical Field

[0001] This invention belongs to the field of neural network hardware implementation technology, and relates to a circuit and method for implementing a runtime linear approximation activation function based on curvature optimization. Background Technology

[0002] Deep Neural Networks (DNNs) rely on Nonlinear Activation Functions (NAFs) to introduce nonlinear components into the network, thereby modeling the complex relationships between input and output data. The main mathematical forms of activation functions include the Sigmoid, Tanh, ReLU, Leaky ReLU, PReLU, ELU, SELU, Swish, and Mish functions. To accelerate DNN inference, the Neural Processing Unit (NPU) must support the computation of various activation functions, thus requiring dedicated hardware circuitry. The hardware implementation of activation functions must strike a balance between computational accuracy, resource overhead, and function versatility. General-purpose processors, such as Nvidia GPUs, Ascend 310, and AMD Instinct, typically use vector / scalar operation engines to perform real-time calculations directly based on mathematical expressions. While this approach maintains the highest accuracy and the widest range of supported functions, it consumes a significant amount of computation cycles and storage bandwidth, making it difficult to meet the low latency and low power consumption requirements of edge-side NPUs. To achieve lightweight acceleration of edge-side NPUs, existing solutions typically employ look-up tables (LUTs) or piecewise linear (PWL) approximations to approximate arbitrary activation functions. LUTs offer high numerical accuracy but incur significantly higher storage costs than PWL. Given that most network models have a certain tolerance for data precision perturbations, PWL methods are more suitable for edge computing scenarios. However, existing PWL methods often use uniformly spaced segments, failing to fully utilize the nonlinear distribution characteristics of activation functions. While non-uniform segmentation strategies have been explored, consistent approximation accuracy across different activation functions remains difficult to achieve. The upper bound of the error in piecewise approximation is quantitatively related to the local curvature of the objective function; a larger curvature results in more pronounced function bending and a greater error in linear approximation. Therefore, using curvature as an evaluation metric can accurately identify precision-sensitive intervals, enabling precise allocation of limited segmented resources. Developing a curvature-driven automated parameter generation process and a reconfigurable hardware architecture has become one of the current technical challenges to be addressed. Summary of the Invention

[0003] To address the problems existing in the above-mentioned traditional methods, this invention proposes a runtime linear approximation activation function implementation circuit based on curvature optimization and a runtime linear approximation activation function implementation method based on curvature optimization. This can decouple the mathematical expression of the function from the hardware topology and construct a curvature-driven automated parameter generation process and a reconfigurable hardware architecture.

[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a runtime linear approximation activation function implementation circuit based on curvature optimization is provided, including a parameter memory, a parallel comparator array, a thermometer code encoder, and a multiply-accumulate unit. The parameter memory contains target parameters for the target activation function. The target parameters include breakpoint thresholds, slopes, and intercepts generated offline for the target activation function. The target parameters are directly addressed through a lookup table structure and dynamically updated during circuit operation. The parallel comparator array compares the input data to obtain the comparison result represented by thermometer code. The thermometer code encoder encodes the comparison result as input, and the encoded result is the segment to which the input data belongs. The parameter memory reads the slope and intercept of the corresponding segment in the target parameters according to the encoded result and sends them to the multiply-accumulate unit. The multiply-accumulate unit performs multiply-accumulate operations on the input data according to the corresponding slope and intercept, and outputs an approximate result of the target activation function. The process of obtaining the target parameters includes: Based on the computational accuracy of the circuit implemented by the linear approximation activation function during runtime, the input domain of the target activation function is uniformly divided, and the radius of curvature in each interval is calculated; the radius of curvature is taken as the average radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval; Within each segmented interval, calculate the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function; The greedy iterative automatic breakpoint selection algorithm is used to traverse all internal breakpoints based on the divided breakpoints and calculate the target breakpoint that plays a dominant role in the approximate accuracy. Based on the target breakpoint, the initial values ​​of the slope and intercept of each segment are calculated using the least squares method. Then, the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm is used for global iterative optimization to obtain the optimal slope and intercept of each segment.

[0005] In one embodiment, during the acquisition of the target parameters, the initial uniform division is 2 w Section; among which w Specify the input data bit width for the multiplier; The greedy iterative breakpoint automatic filtering algorithm includes the following steps: Determine the range of values ​​for the input data based on the quantization parameters of the input data, and set the parameters within the range.N A uniformly distributed initial breakpoint; N= 2 w +1 (including the endpoints of the value range); Traverse each internal breakpoint within the range of values. i Calculate the point i -1 and point i The approximation error caused by using a line segment between two points to approximate the original curve of the activation function between +1; 0 <i≤ N -1; After completing one round of traversal, delete the breakpoint with the smallest error. i d ; Repeat the traversal and calculation multiple times until the remainder is reached. n-1 A non-uniform breakpoint is used as the target breakpoint; n The target number of segments; In the process of global iterative optimization using the BFGS algorithm, the parameter vector for the m-th iteration is: , k 1… k n Let be the slope of each segment. b 1… b n The intercepts for each segment; The objective function is the total approximation error of each segment. ;in, This is the sum of squares of the fitting residuals for each segment. This is the sum of the squares of the differences in function values ​​between adjacent segments at their common connection points. This is the penalty coefficient; The operations performed in each iteration include: Calculate search direction ;in, It is an approximation of the inverse Hessian matrix and its initial value is set to the identity matrix. I , For gradient operators; The optimal step size is determined by a one-dimensional linear search. Update parameter vector ; Calculate the change vector ; Update the parameter increment. The gradient change increment; Update the approximate matrix of the inverse Hessian matrix based on the change vector. ; After multiple iterations to bring the objective function to converge, the optimal slope and intercept of each segment with the minimum continuous approximation error at the boundary are finally obtained.

[0006] In one embodiment, the slope of each linear fitting segment is obtained using the least squares method. k and intercept b When the initial value is: ,

[0007] in, and These represent the x and y coordinates of each sampling point within the segment. and This represents the mean coordinates of the sampling points. n The number of target breakpoints.

[0008] In one embodiment, the process of calculating the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function includes: connecting the endpoints of the target activation function. and The obtained line segments l As a linear approximation of segmented intervals; line segment l Maximum vertical deviation from the original target activation function curve The maximum absolute error is considered as a linear approximation. Maximum vertical deviation The calculation formula is:

[0009] in, R Let be the radius of curvature of the interval.

[0010] In one embodiment, the number of comparators in the parallel comparator array is the same as the number of breakpoints.

[0011] In one embodiment, the parameter memory is implemented using a single-port SRAM, and the configuration parameters of the target activation function are written before the neural network inference begins and remain unchanged during the inference process.

[0012] In one embodiment, the thermometer encoder outputs the n-1 bit thermometer code (n=2) from the parallel comparator array. s The input data is encoded as a numerical value with a bit width of s. The numerical value with a bit width of s is used to indicate the segment interval in which the input data is located, and is used to read the intercept and slope of the corresponding segment from the parameter memory and load them into the multiplier and adder to calculate the approximate activation value based on the input data.

[0013] In one embodiment, the target activation function may include, but is not limited to, any one of the following: Sigmoid function, Tanh function, ReLU function, Leaky ReLU function, PReLU function, ELU function, SELU function, Swish function, and Mish function.

[0014] On the other hand, a runtime linear approximation activation function implementation method based on curvature optimization is also provided, including the following steps: Obtain the target parameters of the target activation function; the target parameters include the breakpoint threshold, slope, and intercept generated offline for the target activation function. The target parameters are written to the parameter memory during circuit runtime via a lightweight configuration interface; the target parameters are directly addressed through a lookup table structure and dynamically updated during circuit runtime. The parallel comparator array compares and processes the input data to obtain the comparison result represented in thermometer code; The thermometer code encoder uses the comparison result as input for encoding; the encoding result is the segment to which the input data belongs. The parameter memory reads the slope and intercept of the corresponding segment in the target parameters according to the encoding result and then sends them to the multiply-accumulate unit; The multiply-accumulate unit performs multiply-accumulate operations on the input data according to the corresponding slope and intercept, and outputs an approximate result of the target activation function; The process of obtaining the target parameters includes: Based on the computational accuracy of the circuit implemented by the linear approximation activation function during runtime, the input domain of the target activation function is uniformly divided, and the radius of curvature in each interval is calculated; the radius of curvature is taken as the average radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval; Within each segmented interval, calculate the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function; The greedy iterative automatic breakpoint selection algorithm is used to traverse all internal breakpoints based on the divided breakpoints and calculate the target breakpoint that plays a dominant role in the approximate accuracy. Based on the target breakpoint, the initial values ​​of the slope and intercept of each segment are calculated using the least squares method. Then, the BFGS algorithm is used for global iterative optimization to obtain the optimal slope and intercept of each segment.

[0015] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned runtime linear approximation activation function implementation circuit and method based on curvature optimization is implemented in hardware by constructing a runtime linear approximation activation function implementation circuit composed of a parameter memory, a parallel comparator array, a thermometer code encoder, and a multiply-accumulate unit. During circuit operation, the target parameters are written into the parameter memory to switch the circuit to the current hardware implementation of the target activation function in real time. Specifically, when estimating the PWL approximation error, the average radius of curvature within a small interval is calculated to determine the upper bound of the maximum absolute error of the linear approximation. This upper bound is then used to select breakpoints, significantly reducing the average absolute error compared to uniform piecewise PWL. Subsequently, a greedy iterative breakpoint automatic selection algorithm is used to traverse and calculate the target breakpoints that play a dominant role in approximation accuracy, significantly reducing computational complexity while accurately tilting piecewise resources towards high curvature regions. When solving for the slope and intercept of each linear segment, a two-stage solution strategy is adopted, considering the continuity of the piecewise linear function and effectively overcoming the limitations of independent fitting of a single segment, outputting an optimal parameter set that balances mathematical accuracy and hardware fixed-point quantization requirements. Furthermore, the hardware circuit is based on a parallel comparator and supports runtime configurable design through a lightweight configuration interface. It supports dynamic refreshing of on-chip parameters without interruption of the inference data stream, decoupling the mathematical expression of functions from the hardware topology, and constructing a curvature-driven automated parameter generation process and a dynamically configurable hardware architecture. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the hardware architecture of a circuit implementing a runtime linear approximation activation function based on curvature optimization in one embodiment; Figure 2 This is a schematic diagram of the overall architecture of the technical solution adopted by the present invention in one embodiment; Figure 3 This is a schematic diagram illustrating curvature-related calculations in one embodiment; wherein, Figure 3 (a) is the mean radius of curvature. Figure 3 (b) represents the error based on curvature; Figure 4 This is a flowchart illustrating the greedy iterative breakpoint automatic filtering algorithm in one embodiment; Figure 5 This is a block diagram of a reconfigurable PWL hardware circuit implementation in one embodiment; Figure 6This is a flowchart illustrating a method for implementing a runtime linear approximation activation function based on curvature optimization in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0019] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0020] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] To balance lightweight design and real-time performance, existing NPU activation function hardware implementations are mainly divided into three categories. The first category is dedicated circuit implementations: these solutions customize hardware architectures for specific activation functions (such as the Sigmoid and Tanh functions), reusing core computational units (such as exponential operation circuits) by exploiting mathematical similarities between functions, and deriving the target result using basic operations such as addition, subtraction, and shifting. This approach achieves extremely high area efficiency and data throughput for the target function, but the circuit structure is highly rigid, making it difficult to adapt to new and complex activation functions such as Swish and GELU, thus limiting its versatility. The second category is iterative algorithms and general-purpose kernel solutions: these solutions employ general-purpose nonlinear function computation kernels combined with iterative convergence algorithms such as the CORDIC and Newton-Raphson algorithms. This approach sacrifices pipeline delay and hardware control logic complexity to gain support for a wider range of function families.

[0022] The third category comprises lookup table and piecewise linear approximation schemes: These schemes use lookup table (LUT) or piecewise linear (PWL) functions to approximate arbitrary activation functions, balancing the dual requirements of computational efficiency and versatility at the edge. The LUT method discretizes the input domain and pre-stores the function values ​​in on-chip memory, directly reading the output via address indexing; however, its storage resource overhead increases exponentially with increasing accuracy. In contrast, the PWL approximation method fits a nonlinear curve to a linear segment, exhibiting superior area efficiency. Given the robustness of DNNs to small changes in activation function accuracy, the PWL method is more suitable for activation function implementation on edge NPUs.

[0023] The performance of PWL approximation depends primarily on the segmentation strategy and the parameter settings for each segment. Regarding segmentation strategies, while basic PWL methods achieve a simple hardware structure by dividing the input domain into equal intervals, their accuracy is suboptimal because they ignore the nonlinear distribution characteristics of the activation function. To improve accuracy, advanced PWL methods employ non-uniform segmentation and adaptive breakpoint selection strategies: manual segmentation, while considering circuit compatibility and relying on experience to select breakpoints, is cumbersome and struggles to guarantee global optimality; automatic segmentation methods determine segment intervals through affine transformations and automatically select breakpoints based on statistical analysis and iterative optimization, striving for a uniform distribution of error across segments, but lack a quantitative mapping relationship between function nonlinearity and approximation error. In parameter calculation, existing schemes often use least squares (OLS) or neural networks to assist in optimizing the slope and intercept, but generally lack a global optimization mechanism that considers the continuity constraints of the segments and struggles to dynamically configure parameters on the hardware side.

[0024] Based on the preceding analysis of existing technologies, the existing PWL approximation method has three objective drawbacks in the hardware implementation of activation functions: mathematical modeling, automatic parameter optimization, and circuit structure. These are as follows: (1) The lack of quantitative mapping between the nonlinear characteristics of the function and the PWL approximation error in the selection of breakpoints makes it impossible for the distribution of breakpoints to accurately match the changing trend of the function, resulting in suboptimal approximation accuracy; (2) The distribution of breakpoints and the calculation of parameters of each segment mostly rely on manual optimization based on design experience. The slope and intercept obtained often do not meet the continuity constraint of the connection between the beginning and end of adjacent segments, making it difficult to simultaneously achieve automatic selection of breakpoints and minimization of global error; (3) The existing hardware architecture has high interval positioning delay and insufficient dynamic reconfigurability, making it difficult to meet the dual requirements of low power consumption and high versatility of the edge NPU.

[0025] One existing hardware implementation of the Sigmoid / Tanh function based on regression trees and PWL approximation uses a regression tree to partition the objective function and manually selects 16 nodes from 31 generated candidate nodes as segmentation boundaries. This segmentation logic relies primarily on heuristic statistics of data distribution and lacks a quantitative model between local curvature changes and the maximum error of linear fitting. Although some automation is introduced into the segmentation process, it still ultimately relies on manual tuning and least squares to determine the PWL parameters. Furthermore, while this circuit has low resource overhead, it lacks a dynamic parameter loading mechanism and a general interval indexing architecture due to its deep customization for the Sigmoid and Tanh functions. Therefore, existing solutions struggle to balance high accuracy, strong versatility, and low-overhead dynamic switching. There is an urgent need to decouple the mathematical expression of the function from the hardware topology and construct a curvature-driven automated parameter generation process and a reconfigurable hardware architecture.

[0026] To address the aforementioned shortcomings, this invention proposes the following three improvements: Improvement 1 uses curvature as a quantitative indicator to accurately characterize the nonlinearity of the objective function; Improvement 2 automatically iterates and filters breakpoints based on curvature analysis, and combines the BFGS algorithm to achieve global parameter optimization; Improvement 3 uses a reconfigurable parallel comparator array and a thermometer code encoder to construct a PWL circuit in the hardware architecture.

[0027] Specifically, Improvement 1 establishes a quantitative relationship between PWL approximation error and function curvature, enabling the breakpoint selection process to accurately assess the impact of each breakpoint on the overall accuracy; Improvement 2 realizes the joint optimization of automatic breakpoint screening and segmentation parameters, effectively overcoming the limitations of manual tuning; Improvement 3 endows the hardware module with on-chip parameter dynamic configuration capabilities, achieving single-cycle low-latency operation with low resource overhead, comprehensively improving the circuit's versatility and execution efficiency.

[0028] In one embodiment, such as Figure 1 As shown, a circuit for implementing a runtime linear approximation activation function based on curvature optimization is provided, including a parameter memory MEM, a parallel comparator array CMP, a thermometer code encoder ENC, and a multiply-accumulate unit MAC. The parameter memory MEM contains target parameters for the target activation function. The parameter memory MEM can integrate a lightweight configuration interface, which is used to write the target parameters into the parameter memory MEM during circuit runtime. The target parameters include breakpoint thresholds, slopes, and intercepts generated offline for the target activation function. The target parameters are directly addressed through a lookup table structure and dynamically updated during circuit runtime.

[0029] The parallel comparator array (CMP) compares the input data to obtain the comparison result represented by thermometer code. The thermometer code encoder (ENC) encodes the comparison result as input, and the encoded result is the segment to which the input data belongs. The parameter memory (MEM) reads the slope and intercept of the corresponding segment in the target parameters according to the encoded result and sends them to the multiply-accumulate unit (MAC). The multiply-accumulate unit (MAC) performs multiply-accumulate operations on the input data according to the corresponding slope and intercept, and outputs an approximate result of the target activation function.

[0030] The process of obtaining the target parameters may include the following steps: Based on the computational accuracy of the linear approximation activation function implementation circuit, the input domain of the target activation function is uniformly divided, and the radius of curvature in each interval is calculated. The radius of curvature can be the average radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval. Within each segmented interval, the upper bound of the maximum absolute error between the linear fitting segment and the original target activation function is calculated. Using a greedy iterative breakpoint automatic selection algorithm, based on the divided breakpoints, all internal breakpoints are traversed to calculate the target breakpoint that plays a dominant role in the approximation accuracy. Based on the target breakpoint, the initial values ​​of the slope and intercept of each segment are calculated using the least squares method. Then, the BFGS algorithm is used for global iterative optimization to obtain the optimal slope and intercept of each segment.

[0031] It is understandable that the overall design and implementation framework of the above-mentioned circuit based on curvature optimization and runtime linear approximation activation function can be as follows: Figure 2 As shown, the framework adopts a collaborative design process of offline software parameter generation and online hardware dynamic reconfiguration, which fully covers the entire chain from function feature extraction, approximation error analysis, non-uniform breakpoint screening, parameter calculation to reconfigurable hardware implementation. It can implement any mainstream activation function on-chip based on non-uniform segmented PWL.

[0032] Specifically, the framework can include five processing stages.

[0033] In Phase 1, the data bit width of the circuit is implemented based on the aforementioned curvature-optimized runtime linear approximation activation function. w (The value can be 8 or 16, but is not limited to), uniformly divide the input domain of the target activation function. Furthermore, the number of uniformly divided intervals can be 2. w The average radius of curvature (MCR) within each interval is calculated to quantify the local nonlinearity of the function in different input intervals.

[0034] It is important to note that the design of Phase 1 differs from a certain existing technology. This embodiment does not pre-determine a fixed number of segments (e.g., 16), but instead uses high-density sampling to provide a basis for subsequent adaptive breakpoint selection. Furthermore, this embodiment selects 2 based on the data bit width of the hardware circuit. wThe single breakpoint ensures that the hardware can accurately represent the breakpoint coordinates, while another existing technology uses the first derivative (slope) as the indicator for breakpoint optimization, which is fundamentally different from the design of this embodiment.

[0035] In Phase Two, for the initially divided intervals from Phase One, based on the inherent average radius of curvature of the original target activation function and the distance between the interval endpoints, the upper bound of the maximum absolute error between the linear fitting segment and the original target activation function is calculated. This establishes a geometric mapping relationship between curvature, interval length, and approximation error within each interval. A certain existing technology uses average curvature to determine whether the error after merging segment points will reach a threshold, relying on empirical formulas (K1, K2). This embodiment employs a completely different design: combining the average radius of curvature and the geometric relationship between the interval endpoints to calculate the upper bound of the error, achieving a quantitative assessment of the importance of breakpoints, entirely determined by the inherent geometric characteristics of the function. Furthermore, another existing technology uses the first derivative (slope) as an indicator for breakpoint optimization, which is fundamentally different from the design of this embodiment.

[0036] In phase three, a greedy iterative breakpoint automatic filtering algorithm was designed, based on the 2 obtained in phase one. w Based on the existing breakpoints, iterate through all internal breakpoints and calculate the deletion of each breakpoint. i The breakpoint with the smallest possible error increment is removed in each iteration. i d Ultimately, the one that plays a dominant role in approximate accuracy is retained. n A target breakpoint can generally be selected. The encoded output bit width is s 0 =s , s It is an integer.

[0037] A current technology employs a breakpoint merging strategy that starts from low-curvature regions and has a fixed merging direction, which may cause redundant breakpoints in high-curvature regions to be missed. This embodiment uses a greedy iterative automatic breakpoint filtering algorithm that deletes breakpoints that minimize the error increment based on the estimated error value. It does not preset the deletion direction; each iteration deletes the optimal breakpoint, and it can remove redundant breakpoints in high-curvature regions. In other words, this embodiment uses a completely different breakpoint filtering method from other schemes and achieves better optimization results.

[0038] In Phase Four, based on the findings of Phase Three... nFor each target breakpoint, a two-stage optimization strategy is employed, combining least squares calculation with global iterative optimization using the BFGS algorithm. First, the initial value of the slope (thi) for each segment is calculated using least squares. Then, the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm is introduced for global iterative optimization. The optimization objectives are minimizing the approximation error and maintaining the function value at the breakpoint. While preserving the continuity of the piecewise linear function, the optimal slope and intercept for each segment are determined. This embodiment employs a unique two-stage optimization strategy, completely different from the linear / nonlinear approximation divide-and-conquer strategy used in some existing technologies for different types of segmentation.

[0039] In stage five, the breakpoint thresholds, slopes, and intercepts generated offline are dynamically loaded into the on-chip parameter memory via a lightweight configuration interface. The input data to be processed is processed by the parallel comparator array (CMP), and the comparison results, represented in thermometer code, are translated into segmented indices by the thermometer code encoder. Subsequently, the circuit uses these indices to address and read the corresponding slope and intercept from the target parameters in the parameter memory to drive the multiply-accumulate unit (MAC) to output a linear approximation result in real time. This framework eliminates the multi-cycle delay of traditional serial or tree-structured indexes and enables runtime parameter configuration.

[0040] In practical deployments, the above-mentioned curvature-optimized runtime linear approximation activation function implementation circuit design does not require modification of the hardware circuit structure. It only requires updating the register parameters of the on-chip parameter memory MEM through a lightweight configuration interface to achieve dynamic switching of multiple mainstream activation functions on the same hardware circuit. This implementation circuit can achieve high-precision approximation of almost all activation functions.

[0041] Although a certain prior art employs a similar on-chip parameter storage + comparator + multiply-accumulate calculation mode to this embodiment (i.e., a necessary component of the PWL approximation circuit), it lacks the aforementioned circuit and application-level optimizations. This embodiment uses a parallel comparator array to avoid the multi-cycle delay of serial or tree-based indexing, and the thermometer encoder uses a middle bit lookup method for fast single-cycle processing of the parallel comparator output. It also features a dynamically configurable memory interface, enabling rapid reloading of parameters such as slope and intercept during runtime, achieving dynamic switching of the activation function. Another prior art employs a nonlinear approximation / linear approximation divide-and-conquer hardware design strategy; its linear approximation part also lacks special optimization and dynamic parameter configuration.

[0042] The aforementioned runtime linear approximation activation function implementation circuit based on curvature optimization is implemented in hardware by constructing a runtime linear approximation activation function implementation circuit composed of a parameter memory (MEM), a parallel comparator array (CMP), a thermometer code encoder (ENC), and a multiply-accumulate unit (MAC). During circuit operation, the target parameters are written to the parameter memory through a lightweight configuration interface to switch the circuit to the current hardware implementation of the target activation function in real time. Specifically, when estimating the PWL approximation error, the average radius of curvature within a small interval is calculated to determine the upper bound of the maximum absolute error of the linear approximation. This upper bound is then used to select breakpoints, significantly reducing the average absolute error compared to uniform piecewise PWL. Subsequently, a greedy iterative breakpoint automatic selection algorithm is used to traverse and calculate the target breakpoints that play a dominant role in approximation accuracy, significantly reducing computational complexity while accurately tilting piecewise resources towards high curvature regions. When solving for the slope and intercept of each linear segment, a two-stage solution strategy is adopted, considering the continuity of the piecewise linear function and effectively overcoming the limitations of independent fitting of a single segment, outputting an optimal parameter set that balances mathematical accuracy and hardware fixed-point quantization requirements. Furthermore, the hardware circuit is based on a parallel comparator and supports runtime configurable design through a lightweight configuration interface. It supports dynamic refreshing of on-chip parameters without interruption of the inference data stream, decoupling the mathematical expression of the function from the hardware topology, and constructing a curvature-driven automated parameter generation process and a reconfigurable hardware architecture.

[0043] Furthermore, the thermometer encoder can convert the n-1 bits of thermometer code output by the parallel comparator array (n=2) s The value is encoded as a bit width of s; the bit width of s can be used to accurately indicate the segment interval (i.e., segment index) where the input data is located, and can be directly used to read the intercept and slope of the corresponding segment from the parameter memory and load them into the multiplier and adder for corresponding calculations.

[0044] The parameter memory can be implemented using, but is not limited to, a single-port SRAM. The configuration parameters of the target activation function can be dynamically updated, but should be written before the neural network inference begins and remain unchanged during inference, thereby enabling the target activation function to be approximated at a lower hardware cost. It is understood that in other embodiments, for example, a dual-port SRAM can also be used to implement the above-mentioned parameter memory, as long as the required functions can be reliably achieved.

[0045] More specifically, such as Figure 2 As shown, PWL error analysis based on curvature: In the piecewise linear (PWL) approximation, the selection of breakpoints is crucial for optimizing the error distribution. The magnitude of the approximation error depends simultaneously on the segment length and the local curvature of the objective function, where curvature provides an effective measure for error assessment across segments. For the objective function... Its radius of curvature With curvature Defined as: (1) Compared to the instantaneous radius of curvature, the mean radius of curvature (MCR) can more accurately characterize the nonlinearity of the activation function within each segment. For example... Figure 3 As shown in (a), given arc rotation angle Arc length and instantaneous radius of curvature Its average radius of curvature R The calculation is as follows: (2) By combining the average radius of curvature and the line segment length, an accurate estimate of the approximate error can be made. In the process of calculating the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function, such as... Figure 3 (b) shows the assumption that the target activation function is at the endpoint coordinates. (i.e., point P1) ) and coordinates (i.e., point P2) The interval between ) is formed by line segments l Approximately, the line segment l Maximum vertical deviation from the original target activation function curve This can be considered as the upper bound of the maximum absolute error for a linear approximation. d is the chord length; the maximum perpendicular deviation... The calculation formula is: (3) This error increases with increasing segment length and curvature, consistent with the nonlinear distribution characteristics of the activation function. The upper bound of this error limits the maximum range of the fitting error, providing an effective quantitative metric for automatically selecting key breakpoints that significantly impact accuracy. (Radius of curvature) R The mean radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval can be used.

[0046] In one embodiment, during the acquisition of the target parameters, the initial uniform division is 2 w Section; among which w The input data bit width of the multiplier (related to the calculation accuracy of the circuit).

[0047] The greedy iterative automatic breakpoint selection algorithm may specifically include the following processing steps: Determine the range of values ​​for the input data based on the quantization parameters of the input data, and set the parameters within the range. N A uniformly distributed initial breakpoint; N= 2 w +1; Traverse each internal breakpoint within the range of values. i Calculate the point i -1 and point i The approximation error caused by using a line segment between two points to approximate the original curve of the activation function between +1; 0<i ≤ N -1; After completing one round of traversal, delete the breakpoint with the smallest error. i d ; Repeat the traversal and calculation multiple times until the remainder is reached. n-1 A non-uniform breakpoint is used as the target breakpoint; n The target number of segments is determined. After determining the segmentation points, the initial values ​​of the slope and intercept of each linear fitting segment can be obtained by using the least squares method. Then, the BFGS algorithm is used to carry out global iterative optimization, so as to obtain the optimal fitting parameters while ensuring the strict continuity of the boundaries of each segment.

[0048] It is understandable that the settings are within the range of values ​​for the input data. N 1. Initial breakpoints are uniformly distributed; traverse each internal breakpoint. i Calculate the direct connection with the point i Adjacent points i -1 and point i +1 introduces an approximate error; 0< i ≤ N -1; After completing one round of traversal, delete the breakpoint with the smallest approximate error. i d Repeat the traversal and calculation multiple times until the remaining... n-1 A non-uniform breakpoint is used as the target breakpoint; n The target number of segments.

[0049] Specifically, this embodiment is designed as follows: Figure 4 The greedy iterative automatic breakpoint selection algorithm shown below has the following process: (a) Determine the range of values ​​for the input data based on the quantization parameters of the input data, and set the range of values ​​within the specified range. N A uniformly distributed initial breakpoint; N= 2 w +1 (including the endpoints of the value range), where w Specify the input data bit width for the multiplier; (b) Traverse each internal breakpoint i (0< i ≤ N -1), calculate the direct connection points. i Adjacent points i -1 and point iThe approximate error generated between +1; where the error can be calculated by referring to the maximum vertical deviation. The calculation method is the same; (c) After completing one round of traversal, delete the breakpoint with the smallest approximate error. i d ; (d) Repeat steps (b) and (c) until the remainder is reached. n There are non-uniform breakpoints.

[0050] This algorithm only retains parameters that play a dominant role in approximation accuracy. n The algorithm uses a predefined set of non-uniform breakpoints. After the first iteration, the algorithm only needs to recalculate the points that were deleted. i d The error between adjacent regions reduces the overall computational complexity to [a certain level]. At the hardware implementation level, this mechanism ensures that the total number of segments remains fixed before and after filtering. The circuit only needs to be reconfigured according to the newly generated parameters without changing the number of segments or reconstructing the comparator array topology, thus balancing algorithm automation and hardware implementation stability.

[0051] The result was obtained through a greedy iterative breakpoint automatic filtering algorithm. n After a non-uniform breakpoint, this embodiment employs a two-stage optimization strategy to calculate parameters such as the slope and intercept of each segment. Utilizing segmental continuity constraints, the slope of all linear segments can be derived sequentially from the leftmost breakpoint. Each linear segment can be represented by equation (4), where, This represents the piecewise linear approximation function value. k and b These represent the slope and intercept of each segment.

[0052] (4) A two-stage optimization strategy is used to calculate the slope. k and intercept b In the initial value calculation stage, for each linear segment, the slope of each linear fitting segment is obtained using the least squares method. k and intercept b When the initial value is given, it is as shown in equation (5). and These represent the x and y coordinates of each sampling point within the segment. n The number of target breakpoints.

[0053] (5) , and This represents the mean coordinates of the sampling points. Since the starting point coordinates of each segment are fixed due to the constraint of the ending point of the previous segment, relying solely on the least squares method cannot achieve global optimization. Therefore, in the global optimization stage, this embodiment introduces the BFGS algorithm to iteratively optimize the calculation results of the least squares method.

[0054] Furthermore, during the global iterative optimization process using the BFGS algorithm, the parameter vector for the m-th iteration can be set as follows: ,, k 1… k n Let be the slope of each segment. b 1… b n Let be the intercept of each segment, and let the objective function be the total approximation error of each segment. ;in, This is the sum of squares of the fitting residuals for each segment. This is the sum of the squares of the differences in function values ​​between adjacent segments at their common connection points. The continuity constraint is embedded in the objective function of the BFGS algorithm as a penalty function, with the penalty coefficient as the penalty coefficient. Perform the following operations in each iteration: Calculate search direction ;in, It is an approximation of the inverse Hessian matrix and its initial value is set to the identity matrix. I , For gradient operators; Determine the optimal step size using a one-dimensional line search. Update parameter vector ; Calculate the change vector ; Update the parameter increment. The gradient change increment; Based on the change vector, update the approximate matrix of the inverse Hessian matrix as follows: (6) To strictly guarantee the piecewise continuity of PWL, the continuity constraint is embedded as a penalty function into the objective function of the BFGS algorithm as follows: (7) in, i For the index of the data sampling points, j For breakpoint index, For gradient operators, M This represents the total number of global sampling points. It is a piecewise linear approximation function. This is the penalty factor. After multiple iterations to bring the objective function to convergence, the optimal slope and intercept of each segment with the minimum continuous approximation error at the boundary are finally obtained.

[0055] The BFGS algorithm updates the parameter vector iteratively. and its inverse Hessian matrix approximation While minimizing the approximation error, a penalty term is used. By forcing the function value difference between adjacent segments at the breakpoint to approach zero, global optimization is achieved while strictly maintaining the continuity of PWL segments, and the optimal slope and intercept of each segment are finally calculated.

[0056] The two-stage optimization strategy described above effectively overcomes the limitations of single-segment independent fitting, outputting an optimal parameter set that balances mathematical accuracy and hardware fixed-point quantization requirements, which can be directly written into the on-chip parameter memory of the reconfigurable circuit.

[0057] In one embodiment, the target activation function may be, but is not limited to, any one of the following: Sigmoid function, Tanh function, ReLU function, Leaky ReLU function, PReLU function, ELU function, SELU function, Swish function, and Mish function.

[0058] In one embodiment, the number of comparators in the parallel comparator array is the same as the number of breakpoints.

[0059] It is understandable that the reconfigurable hardware circuit design is as follows: Through the aforementioned optimization design, the breakpoint threshold th and slope of any nonlinear activation function can be determined. k With intercept b All can be generated offline. Figure 5 The paper presents a specific hardware implementation scheme for a runtime linear approximation activation function based on curvature optimization. This scheme adopts a reconfigurable design and can dynamically switch between different types of activation functions by updating parameters such as breakpoint threshold, slope and intercept stored in the parameter memory in real time during runtime.

[0060] like Figure 5 As shown, input data x First, the comparison result is processed by a parallel comparator array (CMP) to obtain an n-1 bit thermometer code representation (where n is the number of segment intervals). This comparison result is then input as thermometer code to the thermometer code encoder (ENC), where it is encoded into a value with a bit width of s. The bit width of s indicates the segment interval in which the input data is located, and is used to read the intercept and slope of the corresponding segment from the parameter memory, which are then loaded into the multiplier and adder. For example, [the following is an example of a process]. The calculations ultimately output an approximate result of the activation function.

[0061] In one embodiment, the parameter memory can be implemented using a single-port SRAM, and the configuration parameters of the target activation function can be dynamically updated, but should be written before the neural network inference begins and remain unchanged during the inference process.

[0062] Specifically, to support runtime configuration, this embodiment can also integrate a lightweight configuration interface in the parameter memory to complete SRAM configuration. This lightweight configuration interface may include: an address bus for target register addressing, a data bus for parameter sending, and a configuration enable signal for coordinating configuration timing. The lightweight configuration interface and the computation pipeline adopt a time-division multiplexing mechanism: when the configuration enable signal is valid, the parameter reading path is temporarily frozen and the parameter writing is completed; after configuration, the computation path is immediately restored, realizing dynamic switching of breakpoint threshold, slope, and intercept, so that the hardware can adapt to different activation functions without interrupting the inference data flow.

[0063] In one embodiment, the approximate error of PWL is derived based on the average radius of curvature of each linear segment. In a practical implementation, if the instantaneous radius of curvature at the midpoint of each linear segment interval is used instead, or the maximum radius of curvature within the segment interval is used directly, the essence of error analysis based on curvature will not be changed, so this is also an alternative solution.

[0064] In one embodiment, in the algorithm for iteratively screening non-uniform breakpoints, it may be required to calculate the values ​​adjacent to the breakpoint. i point i -1 and point i The activation function between +1 uses the approximation error generated by the line segment between two points to approximate the original curve, and deletes the breakpoint with the smallest error after one round of traversal. i d In the specific implementation, if the points are connected respectively... i -1 and point i ,point i and points i +1 calculates the average error of the two linear segments and uses this as the basis for deleting intermediate breakpoints. This does not change the fact that the breakpoints with the least impact on the error are deleted iteratively, so this is also an alternative solution.

[0065] Compared with existing technical solutions, the above design effectively improves the approximation accuracy through a non-uniform segmentation strategy based on curvature. Compared to the traditional PWL method, which suffers from resource waste in smooth regions and excessive error in sharp regions, this invention derives an upper bound for linear approximation based on the average radius of curvature, and uses this as a basis for breakpoint selection. This mechanism allows the segment boundaries to conform to the local curvature changes of the activation function, optimizing the breakpoint distribution with a fixed number of segments. Tests show that, in a 16 / 32-segment configuration, the design of this invention reduces the average absolute error by 64.4% compared to uniform segmentation PWL; in a 32-segment configuration, the maximum absolute error is reduced by an average of 44.0% compared to the existing symmetric sensing non-uniform segmentation architecture (X. Feng, ICFPT'2021), demonstrating superior accuracy compared to similar solutions.

[0066] In existing PWL frameworks, breakpoint distribution relies on experience or heuristic rules, making it difficult to accurately match local nonlinear characteristics of the function and limiting the approximation accuracy with a fixed number of segments. To address this, this invention proposes a greedy iterative automatic breakpoint selection algorithm based on curvature analysis: using the upper bound of the linear approximation error derived from the radius of curvature as a quantitative indicator, and based on uniformly initialized breakpoints, it automatically eliminates low-contribution redundant points through an iterative strategy, retaining only the key breakpoints that dominate the accuracy. The computational complexity of this algorithm is only [missing information]. This enables precise tilting of segmented resources towards high curvature regions. After the breakpoints are determined, the hardware-friendly slope and intercept parameters can be efficiently generated by using the least squares initial value solution and the global continuity optimization of the BFGS algorithm.

[0067] Existing PWL hardware often relies on cascaded serial comparators, consuming multiple clock cycles and involving complex control logic, making it difficult to meet the high throughput requirements of edge NPUs. This invention employs a parallel comparator array and a runtime-configurable on-chip parameter memory. Input data is compared in a single cycle and then mapped to a segment index via a thermometer encoder, eliminating the latency caused by iterative lookups. Combined with a lightweight configuration interface, it supports freezing parameter paths during inference pipeline operation, performing hot refresh of breakpoint thresholds and slope / intercept, and enabling online activation function switching.

[0068] In one embodiment, such as Figure 6 As shown, a runtime linear approximation activation function implementation method based on curvature optimization is also provided, which may include the following processing steps S10 to S20: S10, Obtain the target parameters of the target activation function; the target parameters include the breakpoint threshold, slope and intercept generated offline for the target activation function; S12, the target parameters are written to the parameter memory during circuit operation through the lightweight configuration interface; the target parameters are directly addressed through the lookup table structure and dynamically updated during circuit operation; S14, the parallel comparator array compares and processes the input data to obtain the comparison result represented by the thermometer code; S16, the thermometer code encoder encodes the comparison result as input; the encoding result is the segment to which the input data belongs; S18, the parameter memory reads the slope and intercept of the corresponding segment in the target parameters according to the encoding result and sends them to the multiply-accumulate unit; S20, the multiply-accumulate unit performs multiply-accumulate operations on the input data according to the corresponding slope and intercept, and outputs an approximate result of the target activation function.

[0069] The process of obtaining the target parameters may include: Based on the computational accuracy of the circuit implemented by the linear approximation activation function during runtime, the input domain of the target activation function is uniformly divided, and the radius of curvature in each interval is calculated; the radius of curvature can be the average radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval; Within each segmented interval, calculate the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function; The greedy iterative automatic breakpoint selection algorithm is used to traverse all internal breakpoints based on the divided breakpoints and calculate the target breakpoint that plays a dominant role in the approximate accuracy. Based on the target breakpoint, the initial values ​​of the slope and intercept of each segment are calculated using the least squares method. Then, the BFGS algorithm is used for global iterative optimization to obtain the optimal slope and intercept of each segment.

[0070] The aforementioned method for implementing a runtime linear approximation activation function based on curvature optimization is implemented in hardware using a runtime linear approximation activation function implementation circuit composed of a parameter memory, a parallel comparator array, a thermometer code encoder, and a multiply-accumulate unit. During circuit operation, the target parameters are written to the parameter memory through a lightweight configuration interface, allowing the circuit to switch to the current hardware implementation of the target activation function in real time. Specifically, when estimating the PWL approximation error, the average radius of curvature within a small interval is calculated to determine the upper bound of the maximum absolute error of the linear approximation. This upper bound is then used to select breakpoints, significantly reducing the average absolute error compared to uniform piecewise PWL. Subsequently, a greedy iterative breakpoint automatic selection algorithm is used to traverse and calculate the target breakpoints that dominate the approximation accuracy, significantly reducing computational complexity while accurately tilting piecewise resources towards high curvature regions. When solving for the slope and intercept of each linear segment, a two-stage solution strategy is adopted, considering the continuity of the piecewise linear function and effectively overcoming the limitations of independent fitting of a single segment. This outputs an optimal parameter set that balances mathematical accuracy and hardware fixed-point quantization requirements. The hardware circuit is based on a parallel comparator and supports runtime configurable design through a lightweight configuration interface. It supports dynamic refreshing of on-chip parameters without interruption of inference data flow, decouples the mathematical expression of functions from the hardware topology, and constructs a curvature-driven automated parameter generation process and a reconfigurable hardware architecture.

[0071] It is understood that the explanations of the features in the above-mentioned curvature-optimized runtime linear approximation activation function implementation method can be understood by referring to the corresponding explanations in the various embodiments of the above-mentioned curvature-optimized runtime linear approximation activation function implementation circuit, and will not be repeated here.

[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A circuit for implementing a runtime linear approximation activation function based on curvature optimization, characterized in that, It includes a parameter memory, a parallel comparator array, a thermometer code encoder, and a multiply-accumulate unit. The parameter memory contains target parameters for the target activation function. The target parameters include the breakpoint threshold, slope, and intercept generated offline for the target activation function. The target parameters are directly addressed through a lookup table structure and dynamically updated during circuit operation. The parallel comparator array compares and processes the input data to obtain the comparison result represented by thermometer code. The thermometer code encoder uses the comparison result as input for encoding, and the encoded result is the segment to which the input data belongs. The parameter memory reads the slope and intercept of the corresponding segment in the target parameters according to the encoding result and sends them to the multiply-accumulate unit. The multiply-accumulate unit performs multiply-accumulate operations on the input data according to the corresponding slope and intercept, and outputs an approximate result of the target activation function. The process of obtaining the target parameters includes: Based on the computational accuracy of the circuit implemented by the linear approximation activation function during runtime, the input domain of the target activation function is uniformly divided, and the radius of curvature in each interval is calculated; the radius of curvature is taken as the average radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval; Within each segmented interval, calculate the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function; The greedy iterative automatic breakpoint selection algorithm is used to traverse all internal breakpoints based on the divided breakpoints and calculate the target breakpoint that plays a dominant role in the approximate accuracy. Based on the target breakpoint, the initial values ​​of the slope and intercept of each segment are calculated using the least squares method. Then, the BFGS algorithm is used for global iterative optimization to obtain the optimal slope and intercept of each segment.

2. The circuit for implementing a runtime linear approximation activation function based on curvature optimization according to claim 1, characterized in that, During the acquisition of target parameters, the initial uniform division is 2 w Section, among which w Specify the input data bit width for the multiplier; The greedy iterative breakpoint automatic filtering algorithm includes the following steps: Determine the range of values ​​for the input data based on the quantization parameters of the input data, and set the parameters within the range. N A uniformly distributed initial breakpoint; N= 2 w + 1； Traverse each internal breakpoint within the range of values. i Calculate the point i -1 and point i The approximation error caused by using a line segment between two points to approximate the original curve of the activation function between +1; 0 <i≤ N -1; After completing one round of traversal, delete the breakpoint with the smallest error. i d ; Repeat the traversal and calculation multiple times until the remainder is reached. n-1 A non-uniform breakpoint is used as the target breakpoint; n The target number of segments; In the process of global iterative optimization using the BFGS algorithm, the parameter vector for the m-th iteration is: , k 1… k n Let be the slope of each segment. b 1… b n Let be the intercept of each segment, and let the objective function be the total approximation error of each segment. ;in, This is the sum of squares of the fitting residuals for each segment. This is the sum of the squares of the differences in function values ​​between adjacent segments at their common connection points. This is the penalty coefficient; The operations performed in each iteration include: Calculate search direction ;in, It is an approximation of the inverse Hessian matrix and its initial value is set to the identity matrix. I , For gradient operators; The optimal step size is determined by a one-dimensional linear search. Update parameter vector ; Calculate the change vector ; Update the parameter increment. This represents the gradient change increment; Update the approximate matrix of the inverse Hessian matrix based on the change vector. ; After multiple iterations to bring the objective function to converge, the optimal slope and intercept of each segment with the minimum continuous approximation error at the boundary are finally obtained.

3. The runtime linear approximation activation function implementation circuit based on curvature optimization according to claim 2, characterized in that, The slope of each linear fitting segment is obtained using the least squares method. k and intercept b When the initial value is: ， ； in, and These represent the x and y coordinates of each sampling point within the segment. and This represents the mean coordinates of the sampling points. n The number of target breakpoints.

4. The runtime linear approximation activation function implementation circuit based on curvature optimization according to any one of claims 1 to 3, characterized in that, The process of calculating the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function includes: Connect the target activation function endpoint and The obtained line segments l As a linear approximation of segmented intervals; line segment l Maximum vertical deviation from the original target activation function curve The maximum absolute error is considered as a linear approximation. Maximum vertical deviation The calculation formula is: in, R Let be the radius of curvature of the interval.

5. The circuit for implementing a runtime linear approximation activation function based on curvature optimization according to claim 1, characterized in that, The number of comparators in a parallel comparator array is the same as the number of breakpoints.

6. The runtime linear approximation activation function implementation circuit based on curvature optimization according to claim 1 or 5, characterized in that, The parameter memory is implemented using a single-port SRAM. The configuration parameters of the target activation function are written before the neural network inference begins and remain unchanged during the inference process.

7. The runtime linear approximation activation function implementation circuit based on curvature optimization according to claim 6, characterized in that, The thermometer encoder encodes the n-1 bit thermometer code output by the parallel comparator array into a value with a bit width of s. The value with a bit width of s is used to indicate the segment interval where the input data is located, and is used to read the intercept and slope of the corresponding segment from the parameter memory and load them into the multiplier and adder.

8. The runtime linear approximation activation function implementation circuit based on curvature optimization according to claim 1, characterized in that, The target activation function includes any one of the following: Sigmoid, Tanh, ReLU, Leaky ReLU, PReLU, ELU, SELU, Swish, and Mish.

9. A method for implementing a runtime linear approximation activation function based on curvature optimization, characterized in that, Including the following steps: Obtain the target parameters of the target activation function; the target parameters include the breakpoint threshold, slope and intercept generated offline for the target activation function; write the target parameters to the parameter memory during circuit runtime through a lightweight configuration interface; The target parameters are directly addressed through a lookup table structure and are dynamically updated during circuit operation. The parallel comparator array compares and processes the input data to obtain the comparison result represented in thermometer code; The thermometer code encoder encodes the comparison result as input; The encoding result is the segment to which the input data belongs; The parameter memory reads the slope and intercept of the corresponding segment in the target parameters according to the encoding result and then sends them to the multiply-accumulate unit; The multiply-accumulate unit performs multiply-accumulate operations on the input data according to the corresponding slope and intercept, and outputs an approximate result of the target activation function; The process of obtaining the target parameters includes: Based on the computational accuracy of the circuit implemented by the linear approximation activation function during runtime, the input domain of the target activation function is uniformly divided, and the radius of curvature in each interval is calculated; the radius of curvature is taken as the average radius of curvature, the maximum radius of curvature, or the instantaneous radius of curvature at the midpoint of the interval; Within each segmented interval, calculate the upper bound of the maximum absolute error between the linearly fitted segment and the original target activation function; The greedy iterative automatic breakpoint selection algorithm is used to traverse all internal breakpoints based on the divided breakpoints and calculate the target breakpoint that plays a dominant role in the approximate accuracy. Based on the target breakpoint, the initial values ​​of the slope and intercept of each segment are calculated using the least squares method. Then, the BFGS algorithm is used for global iterative optimization to obtain the optimal slope and intercept of each segment.

10. The method for implementing a runtime linear approximation activation function based on curvature optimization according to claim 9, characterized in that, The parameter memory is implemented using a single-port SRAM. The configuration parameters of the target activation function are written before the neural network inference begins and remain unchanged during the inference process.

Images