Activation function implementation method and device

Abstract

A method and device for implementing an activation function, comprising: obtaining an input value; determining a target segment corresponding to the input value, wherein the target segment is one or more segments of a piecewise linear function; obtaining parameters of the function corresponding to the target segment; calculating a result based on the parameters and the input value; and obtaining an output value based on the calculation result. This method effectively reduces storage overhead and improves computational efficiency.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for implementing an activation function. Background Art

[0002] With the development of society and technology, neural networks have been applied in more and more fields and have made good progress. Activation functions play an important role in neural network calculations.

[0003] An activation function is essentially a nonlinear transformation that maps a neuron's weighted input to its output. The selection and fitting of nonlinear transformation functions has been a crucial research topic throughout the development of information processing and artificial intelligence systems. Early information processing models used simple threshold functions as nonlinear transformation functions. However, with the convergence of information theory and statistical physics, it has been discovered that nonlinear transformation functions are closely related to probability distribution functions in statistical physics.

[0004] Currently, activation functions are generally implemented using activation function approximation methods. This approach accelerates neural network inference computations by approximating complex nonlinear functions such as sigmoid, tanh, and ReLU with simple, hardware-friendly functions. Common approximation methods include polynomial approximation, lookup tables, and quantization. These methods significantly reduce computational and storage overhead while maintaining model accuracy. They can be used in conjunction with techniques such as network pruning and weight quantization to improve model execution efficiency.

[0005] However, the inventors have found through research that the existing activation function implementation method still has the problems of high computational overhead and low efficiency. Summary of the Invention

[0006] The technical problem solved by the present invention is how to solve the problems of high computational overhead and low efficiency in the current activation function implementation method.

[0007] To solve the above technical problems, an embodiment of the present invention provides a method for implementing an activation function, which includes: obtaining an input value; determining a target segment corresponding to the input value, where the target segment is one or more segments in a piecewise linear function; obtaining parameters of the function corresponding to the target segment; calculating a calculation result based on the parameters and the input value; and obtaining an output value based on the calculation result.

[0008] Optionally, determining the target segment corresponding to the input value includes: obtaining the number of segments corresponding to the input value, selecting a target segment based on the number of segments, and the target segment being a segment in a piecewise linear function; obtaining the output value based on the calculation result includes: using the calculation result as the output value.

[0009] Optionally, the target segment is a plurality of segments in the piecewise linear function, and obtaining the output value according to the calculation result includes: selecting one from the obtained calculation results as the output value.

[0010] Optionally, calculation results are obtained by executing the parameters and input values based on different target segments through different thread processing.

[0011] Optionally, the parameters include one or more of the endpoint, slope and offset of this segment of the function.

[0012] Optionally, the slope and / or bias is expressed as a power of 2, where n is represented by a one-hot code.

[0013] Optionally, the method further includes: if the piecewise linear function is an odd function, determining whether the input value is located on the negative half axis; in response to the input value being located on the negative half axis, obtaining the output value based on the calculation result, including: inverting the calculation result to obtain the output value.

[0014] Optionally, the piecewise linear function includes a first segment and a second segment, and the method further includes: obtaining the difference between the parameters of the first segment and the parameters of the second segment; and obtaining the parameters of the second segment based on the parameters of the first segment and the difference.

[0015] In the second aspect, the present application also provides an implementation device of an activation function, including an input end, a target segment determination module, an adder, a memory, a calculator and an output end, wherein: the input end is used to obtain an input value; the target segment determination module is used to determine the target segment corresponding to the input value, and the target segment is one or more segments in a piecewise linear function; the memory is used to store the parameters of each segment function in the piecewise linear function; the calculator is used to calculate the calculation result based on the parameters and the input value; the output end is used to output the output value.

[0016] Optionally, the device also includes a first selector, which is used to obtain the number of segments corresponding to the input value, select a target segment based on the number of segments, and the target segment is a segment in a piecewise linear function; the output value is the calculation result.

[0017] Optionally, the target segment is a plurality of segments in the piecewise linear function, and the device further includes a second selector, which is used to select at least one from the obtained calculation results as the output value.

[0018] Optionally, the calculator includes a shifter and an adder.

[0019] Optionally, the device further includes a negation module, which is used to negate the calculation result to obtain the output value when the piecewise linear function is an odd function and the input value is located on the negative half axis.

[0020] In a third aspect, the present application also provides a field programmable gate array chip, characterized in that the field programmable gate array chip is used to implement the activation function implementation method described in any one of the above items.

[0021] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0022] The implementation method of the activation function provided by the embodiment of the present invention achieves maximum approximation of the activation function while reducing memory usage by designing the segmented intervals of the linear function and selecting specific parameters. The linear piecewise function is a linear function within each segment, and the calculation process only involves simple basic operations such as addition, subtraction, multiplication, and division, avoiding complex nonlinear operations such as exponential, logarithmic, and trigonometric functions, and having higher computational efficiency. The linear piecewise function can be represented by a small number of parameters such as segment endpoints and segment slopes, biases, etc., with very little storage overhead, and there is no need to store a large number of value pairs like a lookup table (LUT).

[0023] Furthermore, after obtaining an input value, the linear function corresponding to the target segment of the input value is first accurately determined. At least one segment of the piecewise linear function is selected as the target segment. In subsequent steps, the corresponding function value is calculated only for the mathematical expression of the selected target segment, resulting in a calculation result that can be directly used as the output value. Thus, by first accurately determining the linear function corresponding to the target segment of the input value, and then performing the function-related calculations to obtain the output value, the computational effort can be effectively reduced.

[0024] Furthermore, before performing function calculations, instead of precisely selecting the target function, the function calculations are performed on multiple target segments, and the output value is finally selected from the function calculation results of the multiple target segments. This allows the most appropriate output result to be selected based on the input value and the multiple calculation results. Furthermore, because each segment of a piecewise linear function is independent, parallel computation of the linear piecewise function using different threads is relatively simple and can significantly improve computational efficiency. Multithreading is also essential in neural network and big data computations.

[0025] Furthermore, for odd functions, it is not necessary to calculate all partitions. After calculating the positive partition, the negative semi-axis can be directly negated to obtain the corresponding function value. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flowchart of a method for implementing a first activation function according to an embodiment of the present invention;

[0027] Figure 2 is a flow chart of a method for implementing a second activation function according to an embodiment of the present invention;

[0028] Figure 3 is a flow chart of a method for implementing a second activation function according to an embodiment of the present invention;

[0029] Figure 4 1 is a structural diagram of a device for implementing the first activation function according to an embodiment of the present invention;

[0030] Figure 5 is a structural diagram of a device for implementing the second activation function according to an embodiment of the present invention;

[0031] Figure 6 It is a structural diagram of a device for implementing the third activation function in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] As mentioned in the background technology, the existing activation function implementation method still has the problems of high computational overhead and low efficiency.

[0033] Specifically, nonlinear function fitting aims to approximate complex nonlinear functions such as sigmoid, tanh, and ReLU using simple, hardware-friendly mathematical functions, reducing computational and storage overhead while maintaining network accuracy. Polynomial approximation utilizes low-order polynomials; look-up tables (LUTs) precompute and store outputs for different inputs. These techniques can be combined with methods such as pruning and quantization to significantly improve the execution efficiency of neural networks.

[0034] Existing hardware still uses the traditional lookup table (LUT) method. In existing hardware implementations, the lookup table (LUT) is a commonly used technology for implementing nonlinear transformation functions (such as activation functions). Essentially, the LUT is a method for precalculating and storing function output values. By dividing the function's input range into several intervals and storing the output values corresponding to each interval in a table, the function is approximated. In many hardware designs, especially in resource-constrained embedded systems and application-specific integrated circuits (ASICs), LUT technology is widely used to implement various nonlinear functions such as sigmoid, hyperbolic tangent (tanh), inverse tangent (arctan), etc.

[0035] After research, the inventors found that the LUT method has the following disadvantages:

[0036] Disadvantage 1: Limited Accuracy. A LUT is a discretized approximation of a function, and its accuracy is limited by the number of table entries and the interpolation method. Increasing the number of table entries improves accuracy, but also increases memory usage. Therefore, there is a trade-off between accuracy and resource usage with a LUT.

[0037] Disadvantage 2: Memory usage. Memory usage is a major limitation of LUT technology, especially when dealing with high-dimensional functions. For an n-dimensional function, if the input range of each dimension is divided into m intervals, the number of entries required to store in the LUT will be m^n. As the dimension n increases, the number of entries increases exponentially. This means that even for a moderately sized LUT, the storage space required can become very large.

[0038] For example, suppose we want to implement a 5-dimensional function, with the input range of each dimension divided into 100 intervals. The number of entries required in the LUT would reach 100^5 = 10,000,000,000 (10 billion). Assuming each entry occupies 4 bytes (single-precision floating-point numbers), the total storage space required for this LUT would reach 40GB. This is a significant overhead, or even unacceptable, for many embedded systems and specialized hardware.

[0039] In addition to the number of table entries, the LUT's bit width (the number of bits each entry occupies) also affects its memory usage. Bit width is generally related to the LUT's precision. If higher precision is required, more bits are needed to represent each function value, further increasing the LUT's memory usage.

[0040] Furthermore, large LUTs not only consume significant storage resources but also impact system power consumption and cost. Large memories typically consume higher power and require more hardware resources to address and read. This is a significant consideration for battery-powered embedded devices or power-sensitive applications.

[0041] Disadvantage 3: Poor scalability. LUTs are designed for specific functions and input ranges. If the function needs to be changed or the input range needs to be expanded, the LUT must be regenerated and stored, which reduces design flexibility and reusability.

[0042] In addition, the inventors have also found through research that the prior art has the following problems:

[0043] Problem 1: Inefficient approximate model fitting speed and energy consumption.

[0044] This is also because the traditional iterative algorithm based on series expansion has the following shortcomings:

[0045] Long latency. First, iterative algorithms typically require multiple iterations to achieve the desired accuracy. More iterations increase the computation time, leading to increased latency. Each iteration requires a series of arithmetic operations and memory accesses, all of which consume clock cycles and, cumulatively, significantly increase the overall latency. In real-time systems or high-speed applications, excessively long iteration latency may not meet timing requirements, impacting system responsiveness and throughput.

[0046] Secondly, series expansion in iterative algorithms typically involves numerous multiplication and addition operations. The execution time of these arithmetic operations directly impacts the latency of each iteration. High-precision arithmetic operations (such as floating-point operations) typically take longer to execute because they require more complex circuit structures and more clock cycles. If specialized arithmetic units such as multipliers and adders are used in the iterative algorithm, their latency characteristics (such as the number of pipeline stages of the multiplier) will also affect the latency of the entire iterative process.

[0047] Furthermore, iterative algorithms frequently read and write intermediate results and status information, which involves accessing on-chip memory (such as SRAM or register files). Memory access time (read and write latency) affects the latency of each iteration, with larger memory capacities typically resulting in longer access times. If the algorithm contains data dependencies (i.e., the current iteration depends on the results of the previous iteration), memory access latency becomes a critical factor affecting the entire iteration process.

[0048] On the other hand, iterative algorithms require additional control logic to manage the iteration process, including loop control, conditional judgment, and state updates. Complex control logic circuits typically require more logic levels and longer combinational logic paths, which increases the propagation delay of control signals. If the control logic delay is large, it will affect the start and completion time of each iteration, and thus the delay of the entire iteration process.

[0049] Finally, hardware implementations of iterative algorithms typically require each iteration to complete within a limited number of clock cycles. The choice of clock frequency directly impacts the latency of the iteration process. A lower clock frequency means each clock cycle lasts longer, allowing more time for arithmetic operations and memory accesses, but it also increases the overall iteration latency. If the iterative algorithm has a long critical path (i.e., a long chain of combinational logic), the clock frequency may need to be reduced to meet timing constraints, further increasing latency.

[0050] In summary, iterative algorithms based on series expansion suffer from significant latency when implemented in hardware due to the high number of iterations, complex arithmetic operations, frequent memory accesses, and control logic overhead. Long iteration latency can affect the system's real-time performance and response speed, limiting the algorithm's use in latency-sensitive applications.

[0051] Each iteration requires multiplication and addition operations, requiring additional multiplier and adder circuits and increasing hardware implementation costs. The iterative process requires additional control logic and storage units to manage state and intermediate results, increasing hardware complexity. Some functions may have multiple fixed points, and the iterative algorithm may converge to an incorrect fixed point, affecting the accuracy of the approximation. Access requires three steps: encoding, storage, and decoding, resulting in high latency and hindering high-speed computation. SRAM or ROM storage is typically used, resulting in large storage unit area and power consumption.

[0052] The second problem is insufficient computing power for complex calculations with higher precision.

[0053] It is not conducive to the calculation of complex functions. As the complexity of the function increases, the required storage space will increase exponentially. It can only map discrete input values. For functions with continuous value ranges, it can only be implemented through interpolation or approximation, which will affect accuracy.

[0054] To solve the above technical problems, first of all, please refer to Figure 1 , an embodiment of the present invention provides a first method for implementing an activation function, the method comprising the following steps S101 to S105, which are specifically described as follows.

[0055] Step S101, obtaining input value;

[0056] Step S102, determining a target segment corresponding to the input value, wherein the target segment is one or more segments in a piecewise linear function;

[0057] Step S103, obtaining parameters of the function corresponding to the target segment;

[0058] Step S104, obtaining a calculation result based on the parameters and input values;

[0059] Step S105: obtaining an output value according to the calculation result.

[0060] Determine the piecewise linear function that the activation function fits, divide the positive half axis of the entire domain of the piecewise linear function into smaller sub-intervals, perform linear fitting operation on the function value of each sub-interval, and obtain the linear function of each segment (or section). The mathematical expression can be expressed as y=ax+b, where the specific values of the relevant coefficients, such as the slope a and the bias (or intercept) b, can be determined by the previous MATLAB fitting simulation, and the coefficients of each linear function segment are stored.

[0061] The input value can be used as the independent variable in a piecewise linear function, and the calculation result is the dependent variable obtained from the mathematical expression based on the piecewise linear function. For example, the mathematical expression of a piecewise linear function is y = 32x + 1024. Taking the input value as x, the calculated y value is the calculation result.

[0062] In Figure 1 the method, by designing the piecewise intervals of the linear function and selecting specific parameters, while achieving the maximum approximation of the activation function, reduces the memory occupancy. The piecewise linear function is a linear function within each section, and the calculation process only involves basic operations such as addition, subtraction, multiplication, and division, avoiding complex non-linear operations such as exponentiation, logarithm, and trigonometric functions, and has higher calculation efficiency. The piecewise linear function can be represented by a small number of parameters such as section endpoints, section slopes, and offsets, with very small storage overhead, and there is no need to store a large number of numerical pairs like a lookup table (LUT).

[0063] After fitting the activation function and implementing it on hardware, due to the simplicity of calculation, the piecewise linear function is easier to implement in a hardware circuit.

[0064] In a specific embodiment, the parameters include one or more of the endpoints, slopes, and offsets of this section of the function. It should be noted that the parameters of the linear function can also include other parameters, which will not be elaborated here.

[0065] This application is mainly a hardware implementation method of a piecewise linear function with extremely low overhead. All operations on the input are implemented with 2^n. For example, for the first piecewise linear function, the mathematical expression is y = 64x, where the value range of x is 0 < x <= 32; for the second piecewise linear function, the mathematical expression is y = 32x + 1024, where the value range of x is 32 < x <= 64; for the third piecewise linear function, the mathematical expression is y = 8x + 2560, where the value range of x is 64 < x <= 196; for the fourth piecewise linear function, the mathematical expression is y = 4095, where the value range of x is x > 196. All operands are 2^n, so that in circuit implementation, only shifting and addition are involved. Compared with the traditional multiplier and adder / lookup table method, the hardware overhead of the dedicated integrated circuit designed by this method is very low. For example, in the first piecewise linear function, the slope is 64, which can be shifted left 4 bits by a shifter. Another example is that in the second piecewise linear function, the slope is 32, which can be shifted left 5 bits by a shifter, and adding 1024 means adding 1 at the 10th bit, and so on. For all activation functions, whether odd or even functions, this method can be used for fitting.

[0066] In a specific embodiment, the slope and / or bias is expressed as 2 to the power of n, where n is represented by a one-hot code. A one-hot code is a code system in which there are as many bits as there are states, and only one bit is 1, while the others are all 0. In the present application, the slope and / or bias of an N-segment linear function can be expressed as 2 to the power of n, where N is the number of one-hot codes, and thus 2^n states of the slope or bias can be represented by N bits of the one-hot code. Thus, the number of bits used to represent the slope and bias can be greatly saved, further saving overhead.

[0067] In one embodiment, see Figure 2 , Figure 2 This is a flow chart of a method for implementing the second activation function of the present application. This method may include the following steps:

[0068] Step S101, obtaining input value;

[0069] Step S201, obtaining the number of segments corresponding to the input value, and selecting a target segment according to the number of segments, wherein the target segment is a segment in a piecewise linear function;

[0070] Step S103, obtaining parameters of the function corresponding to the target segment;

[0071] Step S104, obtaining a calculation result based on the parameters and input values;

[0072] Figure 1 Step S105 in the embodiment may include: step S202, using the calculation result as the output value.

[0073] In this embodiment, after obtaining an input value, the linear function corresponding to the target segment of the input value is first accurately determined. At least one segment of the piecewise linear function is selected as the target segment. In subsequent steps S103 and S104, the function value corresponding to the mathematical expression of only the selected target segment is calculated to obtain a calculation result, which can be directly used as the output value. Thus, by first accurately determining the linear function corresponding to the target segment of the input value, and then performing the function-related calculations to obtain the output value, the amount of computation can be effectively reduced.

[0074] In one embodiment, see Figure 3 , Figure 3 This is a flow chart of a method for implementing the third activation function of the present application. This method may include the following steps:

[0075] Step S101, obtaining input value;

[0076] Step S301, determining a target segment corresponding to the input value, wherein the target segment is a plurality of segments in the piecewise linear function;

[0077] Step S103, obtaining parameters of the function corresponding to the target segment;

[0078] Step S104, obtaining a calculation result based on the parameters and input values;

[0079] Figure 1 Step S105 in the embodiment may include: step S302, selecting one of the obtained calculation results as the output value.

[0080] according to Figure 3 In a specific embodiment, after obtaining an input value, all linear function segments are first used as target segments. Function values for each target segment are calculated to obtain function values (i.e., calculation results) for each function segment. Thus, for the same input value, each function segment based on the piecewise linear function can obtain a calculation result, and the number of calculation results is the same as the number of function segments. After obtaining multiple calculation results, one of the multiple calculation results is selected as the output value based on the input value and each calculation result.

[0081] according to Figure 3 In another specific embodiment, after obtaining an input value, several similar segments are first selected from all linear function segments as target segments. Function values for each target segment are calculated to obtain function values (i.e., calculation results) for each function segment. Thus, for the same input value, multiple target segments based on the piecewise linear function can each obtain a calculation result, and the number of calculation results is the same as the number of target segments. After obtaining multiple calculation results, one of the multiple calculation results is selected as the output value based on the input value and the calculation results.

[0082] In this embodiment, before performing function calculation, a precise target function is not selected. Instead, function calculations are performed on multiple target segments, and then an output value is selected from the function calculation results of the multiple target segments. The most suitable output result can be selected based on the input value and the multiple calculation results.

[0083] Furthermore, calculation results are obtained by executing calculations based on the parameters and input values of different target segments through different thread processing.

[0084] For example, after obtaining the input value, all 16 linear function segments are first used as target segments, and then 16 threads can be called, and each thread performs the calculation work of each linear function segment.

[0085] Because each segment of a piecewise linear function is independent, parallel computation of the linear piecewise function using different threads is relatively simple and can significantly improve computational efficiency. Multithreading is also essential in neural network and big data computations.

[0086] In one embodiment, Figure 1 or Figure 2 or Figure 3 The method may further include: if the piecewise linear function is an odd function, determining whether the input value is located on the negative half axis; in response to the input value being located on the negative half axis, the step S105 obtains the output value according to the calculation result, which may include: inverting the calculation result to obtain the output value.

[0087] For example, the activation function tanh is an odd function. By symmetrically negating the positive interval, it remains valid even when its value is less than 0. For odd functions, there's no need to calculate all partitions. After calculating the positive partition, simply negate the negative semi-axis to obtain the corresponding function value. To conserve hardware, when fitting the activation function as a piecewise linear function, only the positive semi-axis is calculated. For even activation functions, no negation is required. For odd activation functions, a negation calculation is added after the function calculation module.

[0088] In one embodiment, the piecewise linear function includes a first segment and a second segment, and the method further includes: obtaining the difference between the parameters of the first segment and the parameters of the second segment; and obtaining the parameters of the second segment based on the parameters of the first segment and the difference.

[0089] Specifically, the parameters of one segment of a piecewise linear function (such as endpoints, slope, intercept, etc.) can be obtained by taking the parameters of another segment and subtracting it from the difference. This eliminates the need to store the parameters of every linear function segment. Instead, it's sufficient to store the parameters of several segments and their corresponding differences. The parameters of each segment can then be obtained through simple addition or subtraction.

[0090] In a specific embodiment, only the parameters of one segment of the linear function are stored, and the parameters of the other segments of the linear function can be calculated based on the difference.

[0091] In another specific embodiment, for the parameter of the endpoints of each segment function in the piecewise linear function, only the endpoints of one or more segment functions are stored, and the endpoints of the other segment functions are obtained by difference calculation.

[0092] It should be noted that other parameters can also be stored and calculated based on the above method, which will not be described here.

[0093] As a result, the parameters of one segment of a piecewise linear function (such as endpoints, slope, intercept, etc.) can be obtained by taking the difference between the parameters of the other segment and its value, effectively saving storage overhead. However, since some function parameters need to be calculated, some computational efficiency may be sacrificed.

[0094] Regarding the implementation method of the first activation function, this application also provides an implementation device 40 of the first activation function, see Figure 4 The first activation function implementation device 40 includes an input terminal 401, a target segment determination module 402, a memory 403, a calculator 404, and an output terminal 405, wherein: the input terminal 401 is used to obtain an input value; the target segment determination module 402 is used to determine the target segment corresponding to the input value, and the target segment is one or more segments in a piecewise linear function; the memory 403 is used to store parameters of each segment function in the piecewise linear function; the calculator 404 is used to calculate a calculation result based on the parameters and the input value, that is, to perform the calculation result of the linear function (referred to as function calculation); and the output terminal 405 is used to output the output value.

[0095] Optionally, the calculator 404 includes a shifter, and the shifter is used to implement multiplication and division calculations of the linear function.

[0096] Optionally, the calculator 404 includes a shifter and an adder, and the calculation of the linear function is implemented through the shifter and the adder.

[0097] Optionally, the implementation device 40 of the first activation function may further include a negation module, which is used to negate the calculation result to obtain the output value when the piecewise linear function is an odd function and the input value is located on the negative half axis.

[0098] For more information about the working principle and working method of the first activation function implementation device 40, please refer to the above Figure 1 The description of the implementation method of the first activation function in will not be repeated here.

[0099] In one embodiment, the present application also provides a second activation function implementation device, Figure 4 A first selector is added to the activation function implementation device 40, and the first selector is used to obtain the number of segments corresponding to the input value, and select a target segment according to the number of segments, and the target segment is a segment in the piecewise linear function; the output value is the calculation result.

[0100] Specifically, before the calculator performs function calculation, the first selector accurately determines the linear function of the target segment corresponding to the input value, and selects at least one segment from the piecewise linear function as the target segment.

[0101] See Figure 5 , Figure 5A schematic diagram of an embodiment of a device for implementing a second activation function is provided. An input value is fed into the device through an input terminal, and a first selector selects a segment of a linear function as a target segment. Function calculation of the target segment function is then performed using a shift register and an adder, resulting in an output value. This output value is then outputted through an output terminal.

[0102] Optionally, the parameters of the target segment function (including slope and bias) can be stored in a static random access memory (SRAM). After the target segment is determined, the parameters of the target segment function are obtained from the SRAM, and the function calculation of the target segment function is implemented by the shift register and the adder.

[0103] Optionally, if the piecewise linear function is an odd function and the input value is located on the negative half axis, the calculation result needs to be passed through a negation module to obtain the output value.

[0104] The implementation device of the second activation function can be used to execute the implementation method of the second activation function. For the relevant description of the implementation device of the second activation function, please refer to Figure 2 The description of the implementation method of the second activation function is not repeated here.

[0105] In one embodiment, the present application also provides a device for implementing a second activation function, wherein the target segment is a plurality of segments in the piecewise linear function. Figure 4 A second selector is added to the activation function implementation device 40, and the second selector is used to select at least one from the obtained calculation results as the output value.

[0106] Specifically, after obtaining the input value, all linear function segments are first taken as target segments, and the function value of each target segment is calculated to obtain the function value of each function segment (ie, the calculation result).

[0107] See Figure 6 , Figure 6 A schematic diagram of an embodiment of a device for implementing a third activation function is provided. An input value is fed into the device at the input terminal. The device then calculates each piecewise linear function using n segmented calculators (each segmented calculator includes a shift register and an adder for that segment; for example, the calculator for segment 1 includes a shift register and an adder for segment 1), obtaining n calculation results. These n calculation results are then fed into a second selector, and an output value is obtained at the output terminal.

[0108] Figure 6Although the activation function implementation is entirely combinatorial logic, there is still some delay. To ensure output speed, after the input value is entered, the results of all piecewise linear functions are calculated, and then a second selector is used to select which piece of calculation result to output.

[0109] Optionally, if the piecewise linear function is an odd function and the input value is located on the negative half axis, the calculation result needs to be passed through an inversion module and then sent to the second selector.

[0110] Alternatively, the parameters of each segmented linear function can be implemented using corresponding shift registers and registers associated with the adder. When implementing a fixed activation function, such as the tanh function, the tanh function's fitted piecewise linear function can be reused. Therefore, n sets of corresponding calculators can be customized to implement the activation function.

[0111] Optionally, the parameters of the target segment function (including slope and bias) can be stored in a static random access memory (SRAM). The parameters of each segment of the linear function are obtained from the SRAM, and the function calculation of the corresponding function is implemented by the shift register and the adder.

[0112] The implementation device of the third activation function can be used to execute the implementation method of the third activation function. For the relevant description of the implementation device of the third activation function, please refer to Figure 3 The description of the implementation method of the third activation function is not repeated here.

[0113] It should be noted that the activation function implementation devices provided in this application are all implemented through highly coupled hardware and circuits, and a computationally efficient activation function implementation device is provided through simple hardware circuits such as adders, shifters, negation modules, and selectors.

[0114] Furthermore, during application development, the inventors discovered that existing technologies lack hardware-based activation function fitting methods specifically tailored to probabilistic computations. Specifically, probabilistic computations differ from traditional von Neumann architectures in terms of model parameter size and computational complexity, necessitating the design of specialized activation function fitting methods tailored to their specific characteristics. The current lack of hardware accelerators specifically designed for probabilistic computations limits the effective operation and deployment of the Ising model on hardware platforms.

[0115] To solve the above problems, in an application example, the activation function implemented in this application can be used in probabilistic computing, which is an emerging computing paradigm that uses probabilistic models and random processes to provide new ideas and methods for solving combinatorial optimization problems. By mapping the problem into a probabilistic framework and combining it with classic probabilistic graphical models such as the Ising model, probabilistic computing can effectively escape the local optimum and find an approximate optimal solution within a reasonable time. With the development of new technologies such as quantum computing, the application of probabilistic computing in the field of combinatorial optimization has been further expanded, bringing new hope for solving large-scale combinatorial optimization problems. In-depth research and development of probabilistic computing theories and algorithms for combinatorial optimization have important theoretical and practical significance for promoting the development of fields such as operations optimization and artificial intelligence.

[0116] Statistical physics focuses on systems composed of a large number of particles, and describes the macroscopic properties of the system through probability distribution functions. Common probability distribution functions include the Boltzmann distribution, the Fermi-Dirac distribution, the Bose-Einstein distribution, etc. These distribution functions describe the energy state distribution of particles in the system. The sigmoid function commonly used in information processing systems has a similar mathematical form to the Fermi-Dirac distribution. The Fermi-Dirac distribution describes the probability of particles occupying energy levels in a fermion system, and its mathematical form is:

[0117] Where g(ε) is the particle energy, ε f is the chemical potential, k is the Boltzmann constant, and T is the absolute temperature.

[0118] Contrast this with the following mathematical form of the Sigmoid function:

[0119]

[0120] Among them, x is a variable and e is a natural constant.

[0121] It can be seen that the sigmoid function can be regarded as a form of Fermi-Dirac distribution under specific parameters. This similarity inspired people to introduce the concepts of statistical physics into the study of information processing systems.

[0122] By linking nonlinear transformation functions with probability distribution functions, we can leverage the theoretical tools of statistical physics to analyze the behavior and properties of information processing systems. For example, we can analogize the states of elements in a system to the energy states of particles, and the evolution of the system to thermodynamic evolution. This interdisciplinary perspective provides new insights into understanding and optimizing information processing systems.

[0123] Therefore, linking nonlinear transformation functions with probability distribution functions in statistical physics provides a new perspective for theoretical research and practical applications of information processing systems. The activation function implementation scheme provided in this application can also achieve sigmoid function fitting, improving computational efficiency.

[0124] In one embodiment, the present application further provides a field-programmable gate array (FPGA) chip, which is used to implement the activation function implementation method described in any of the above. Since the activation function implementation method provided in this application has a small computational load and a simple structure, it can be deployed in an FPGA chip to implement lightweight activation function operations. The activation function implementation device of this application can be used as part of the hardware structure of the FPGA.

[0125] In one embodiment, the present application also provides a hardware accelerator product structure, specifically including: an FPGA chip: serving as the accelerator's core processor, responsible for performing sparsely connected probability calculations; external memory: used to store model parameters and input data; a controller: used to manage data input and output and schedule the accelerator's operating process; and a PCIe interface: the device exchanges data with other components via the PCIe bus interface.

[0126] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to execute Figures 1 to 3 The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state drive, etc.

[0127] An embodiment of the present invention further provides a computing device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the computer program is executed. Figures 1 to 3 The steps of implementing any activation function in .

[0128] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0129] In the embodiments of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory (volatile tile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiments of the present application may also be a circuit or any other device that can implement a storage function, for storing computer programs and / or data.

[0130] The activation function implementation method provided in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, an SSD).

[0131] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

[0132] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. Any multiple embodiments can be used in combination. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0134] During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software units in the processor. The software unit can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and in combination with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0135] In the embodiments of the present application, the processor of the above-mentioned device may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0136] The present application also provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0138] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0139] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0140] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, server or TRP, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0141] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document indicates that the related objects are in an "or" relationship.

[0142] The term "plurality" used in the embodiments of the present application refers to two or more.

[0143] The first, second, etc. descriptions appearing in the embodiments of this application are only for illustration and distinction of the description objects. There is no order, nor does it indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0144] The "connection" appearing in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0145] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for those skilled in the art, according to the idea of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A device for implementing an activation function, characterized in that: The system comprises an input terminal, a target segment determination module, an adder, a memory, a calculator and an output terminal, wherein: The input terminal is used to obtain an input value; a target segment determination module, configured to determine a target segment corresponding to the input value, wherein the target segment is one or more segments in a piecewise linear function; The piecewise linear function includes a first segment and a second segment, obtaining a difference between a parameter of the first segment and a parameter of the second segment; and obtaining a parameter of the second segment based on the parameter of the first segment and the difference; A memory, used to store parameters of each segment function in the piecewise linear function; A calculator, configured to calculate a result based on the parameters and input values; an output terminal for outputting an output value; the device further includes a first selector, the first selector being configured to obtain a segment number corresponding to the input value, and select a target segment according to the segment number, the target segment being a segment in a piecewise linear function; the output value being the calculation result; The target segment is a plurality of segments in the piecewise linear function. The device also includes a second selector, which is used to select at least one from the obtained calculation results as the output value, specifically including implementing function calculation of each piecewise linear function based on n segmented calculators to obtain n calculation results, and then inputting the n calculation results into the second selector, and obtaining the output value at the output end.

2. The device according to claim 1, characterized in that The calculator includes a shifter and an adder.

3. The device according to claim 1, characterized in that The device further includes a negation module, which is configured to negate the calculation result to obtain the output value when the piecewise linear function is an odd function and the input value is located on a negative semi-axis.

Images