An activation algorithm for end-side models
By adding an upper bound and optimizing the structure of the HardSwish activation function, the HSLU6 activation algorithm is proposed, which solves the problems of large computational cost and accuracy loss of edge models on fixed-point computing devices, and achieves efficient inference accuracy and speed without increasing computing power.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN HAIBEN ELECTRONIC TECH CO LTD
- Filing Date
- 2026-01-29
- Publication Date
- 2026-06-12
AI Technical Summary
Existing edge models suffer from high computational load and severe accuracy loss on fixed-point computing devices, especially when using Swish and HardSwish activation functions, making it difficult to maintain inference accuracy without increasing computing power.
The HSLU6 activation algorithm is adopted. By adding an upper bound and optimizing the function structure on the basis of HardSwish, it adapts to the fixed-point computation characteristics and reduces the computation requirements. The specific formula is f(x) = min(x · ReLU6(x + 3) / 6,6), and the computation is optimized by replacing division with shift operation.
It achieves the same performance as Swish without increasing computing power, and reduces the loss of inference accuracy after model quantization, thus improving the computing speed and accuracy of edge models.
Smart Images
Figure CN122197982A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to artificial neural network data processing methods, and more particularly to an activation algorithm applicable to edge models. Background Technology
[0002] In existing technologies, small models on offline devices are limited by the device's computing resources, and their most prominent characteristic is "smallness," which includes both small model size and low computational cost. Offline devices generally use fixed-point computation, including int8, int16, and even larger int32; the higher the bit count, the greater the computational power required. Activation functions generally have no weights and do not increase model size, mainly affecting computational cost. The computational cost is related to mathematical operations such as addition, subtraction, multiplication, division, exponentiation, and logarithms. Among these, the first three require less computational power, while the latter three require more. Swish is a commonly used activation function that can help models converge faster and better, but swish calculations use exponential operations, which are not suitable. To ensure performance, the industry generally uses HardSwish instead, eliminating the exponential operations.
[0003] In practical applications, while HardSwish significantly reduces computational power requirements compared to Swish, its value range, which has only a lower bound and no upper bound, leads to a very large fluctuation range in the model's weights and activation values. Since edge devices use fixed-point computation, it is difficult to adapt to such large-scale numerical fluctuations. After the floating-point model is quantized into fixed-point, the precision loss accumulates layer by layer throughout the inference process, eventually making the model unusable. Other activation functions with upper and lower bounds need to be used as replacements, such as ReLU6. After quantization, the inference precision loss is controllable, but the performance of floating-point models will be much worse, and it will only be worse after quantization. Summary of the Invention
[0004] The technical problem to be solved by this invention is to provide an activation algorithm for edge models that does not increase computing power and has minimal loss of inference accuracy, in order to address the shortcomings of existing technologies.
[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution.
[0006] An activation algorithm applicable to edge-side models, comprising the HSLU6 function:
[0007] f(x) = min(x · ReLU6(x + 3) / 6, 6);
[0008] The formula for ReLU6 is:
[0009] f(x) = min(max(0, x), 6);
[0010] σ is the sigmoid function, and the formula is:
[0011] In the formula, e is a constant, and the range of z is all real numbers.
[0012] Preferably, the input data of the HSLU6 function is intermediate computation data of the neural network.
[0013] The activation algorithm disclosed in this invention, applicable to edge-side models, adds an upper bound to HardSwish to adapt to edge-side fixed-point computation while maintaining lossless performance. Furthermore, it optimizes the function structure to better suit the characteristics of fixed-point computation, reducing computational power requirements and accelerating computation speed. In practical applications, the activation function HSLU6 proposed in this invention, applicable to small edge-side models, achieves the same performance as HardSwish without increasing computational power, and minimizes inference accuracy loss. Attached Figure Description
[0014] Figure 1 A diagram illustrating the values of the swish, hardSwish, and HSLU6 functions;
[0015] Figure 2 A diagram illustrating the value range of the HSLU6 function;
[0016] Figure 3 A comparison chart of data processing curves for the HardSwish and HSLU6 functions;
[0017] Figure 4 A comparison of curves after adding boundaries to the HardSwish function. Detailed Implementation
[0018] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments.
[0019] This invention discloses an activation algorithm applicable to edge-side models, which includes the HSLU6 function:
[0020] f(x) = min(x · ReLU6(x + 3) / 6, 6);
[0021] The formula for ReLU6 is:
[0022] f(x) = min(max(0, x), 6);
[0023] σ is the sigmoid function, and the formula is:
[0024] In the formula, e is a constant, and the range of z is all real numbers.
[0025] Furthermore, the input data for the HSLU6 function is intermediate computation data from the neural network.
[0026] Example 1
[0027] Please see Figure 1 The existing HardSwish function can fit the Swish function well, but it lacks an upper bound. Therefore, this embodiment adds an upper bound to HardSwish, setting this boundary value to 6. This allows the activation output value to be controlled within a limited range. The improved function, after this setting, can be directly added to the neural network for use; the input and output are the matrix arrays in the neural network.
[0028] The formula for the swish function is: f(x) = x · σ(βx);
[0029] The formula for the Hardswish function is: f(x) = x · ReLU6(x + 3) / 6;
[0030] The formula for the HSLU6 function is: f(x) = min(x · ReLU6(x + 3) / 6, 6);
[0031] The formula for ReLU6 is: f(x) = min(max(0, x), 6);
[0032] σ is the sigmoid function, and the formula is:
[0033] In the formula, e is the natural constant, and the range of z is all real numbers.
[0034] In this embodiment, the HardSwish function removes the exponentiation operation from the swish function but introduces division. Although division requires less computational power than exponentiation, it requires more than addition, subtraction, and multiplication, and can be further optimized. In practical applications, division by 6 can be changed to division by 8, and fixed-point calculations can be performed quickly using bit shifting, further reducing computational power requirements and making calculations faster.
[0035] Based on this embodiment, the formula for the HSLU6 function can be further improved as follows:
[0036] f(x) = min(x·ReLU6(x + 4) / 8, 6); the new function takes the following values: Figure 2 As shown.
[0037] Please see Figure 3Experiments have shown that HSLU6 converges faster and performs better than activation functions such as ReLU6, achieving the same level as Swish. Moreover, compared to the HardSwish function, the inference accuracy loss after model quantization is smaller, achieving the expected goal.
[0038] Please see Figure 4 Comparing hardswish (green), hardswish + boundary 6 (orange), and hardswish + boundary 6 + divided by 8 (blue), it can be seen that the inference process after model quantization suffers less accuracy loss compared to the HardSwish function.
[0039] Compared to existing technologies, this invention adds an upper bound to HardSwish, adapting to edge-side fixed-point computation without performance loss. Furthermore, it optimizes the function structure to better suit the characteristics of fixed-point computation, reducing computational power requirements and accelerating computation speed. In practical applications, the activation function HSLU6 (HardSwish Linear Unit 6) proposed in this invention, suitable for small edge-side models, achieves the same performance as HardSwish without increasing computational power, and with minimal loss of inference accuracy.
[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the technical scope of the present invention should be included within the scope of protection of the present invention.
Claims
1. An activation algorithm applicable to edge-side models, characterized in that, Includes HSLU6 functions: f(x) = min(x · ReLU6(x + 3) / 6, 6); The formula for ReLU6 is: f(x) = min(max(0, x), 6); σ is the sigmoid function, and the formula is: In the formula, e is a constant, and the range of z is all real numbers.
2. The activation algorithm applicable to the end-side model as described in claim 1, characterized in that, The input data for the HSLU6 function is intermediate computation data from the neural network.