A method for improving swish lookup table precision

Abstract

The application provides a method for improving the precision of a Swish lookup table, comprising the following steps: S1, a lookup table is implemented; and S2, the precision of the Swish lookup table is improved. The principle of the lookup table of Swish and the method for improving the precision of the lookup table are described, so as to improve the quantization precision of a model and improve the operation efficiency of the model.

Description

Technical Field

[0001] This invention belongs to the field of neural network optimization technology, and specifically relates to a method for improving the accuracy of Swish lookup tables. Background Technology

[0002] In existing technologies, Swish is an activation function widely used in deep learning models, proposed by Google in 2017. It outperforms ReLU and its variants (such as Leaky ReLU) in certain tasks, especially in deep neural networks. The mathematical expression for Swish is:

[0003] Swish(x)=x*δ(βx) (Formula 1)

[0004] in:

[0005] δ(·) is the sigmoid function:

[0006] Due to the complex calculations of exponential functions, models with Swish activations often run inefficiently. Therefore, to enable models to be deployed quickly and efficiently, Swish is usually used to perform table lookups to approximate the complex float exponential calculations. However, due to the limited number of table entries, the accuracy of the table lookup is not satisfactory.

[0007] In other words, the existing technology has the following drawbacks:

[0008] In the field of model optimization, there is currently no method to improve the accuracy of Swish's active lookup table. Summary of the Invention

[0009] To address the aforementioned issues, the purpose of this application is to propose a method for improving the accuracy of Swish lookup tables. This method explains the principles of Swish lookup tables and methods for improving their accuracy, aiming to enhance the quantization accuracy and operational efficiency of the model.

[0010] Specifically, the present invention provides a method for improving the accuracy of Swish lookup tables, the method comprising the following steps:

[0011] S1. Implementation using a lookup table:

[0012] Determine the main parameters of the lookup table, including the number of entries, the fixed-point width (LBIT) of the input X, and the fixed-point width (TBIT) of the entry storage.

[0013] Divide the X-axis into 32 sampling points and calculate the Y value corresponding to each sampling point;

[0014] Store the quantized output values ​​y0_uint8 and dy_uint8;

[0015] When looking up a table, calculate the coefficient _ceof and the index i, where i is the integer part and _ceof is the decimal part; input x_uint8 and use the formula... Quantification processing is performed;

[0016] S2. Improve the accuracy of Swish lookup tables:

[0017] To address the input characteristics of the Swish function, the input is truncated at -5 before statistical quantization.

[0018] Traverse the computation graph and set the flag is_swish_input for Swish's input;

[0019] When is_swish_input is true, the output of the current layer is truncated at -5 to improve the utilization and accuracy of the lookup table.

[0020] Step S1 further includes:

[0021] S1.1. Input parameter processing: Receive input data x_new and lookup table parameter lut_param, calculate floating-point bit width float_bitwidth, and generate mask mask_float for extracting the decimal part of the input data;

[0022] S1.2. Input data preprocessing: Convert the input data x_new to int32 type and reshape it into a two-dimensional tensor, and calculate the index x_new_indices through a right shift operation;

[0023] S1.3. Lookup table query: Extract the difference table LUT_dy and the base value table LUT_y0 from the lookup table parameter lut_param, use x_new_indices as the index to get the corresponding dy and y0 values, and convert them to float32 type;

[0024] S1.4. Interpolation calculation: Extract the decimal part ceof of the input data x_new, calculate the interpolation result y_new using the linear interpolation formula, and perform fixed-point processing;

[0025] S1.5. Return Result: Output the interpolated result y_new, whose shape is consistent with the input data x_new.

[0026] The lookup table has 32 entries.

[0027] In the table lookup process, the formula for calculating index i is:

[0028]

[0029] Where table_num represents the number of table entries.

[0030] In the quantization process of the input x_uint8, the input bit width in_bit is divided into two parts: table_bit and float_bitwidth.

[0031] The input truncation operation of the Swish function is used to improve the utilization and accuracy of the lookup table.

[0032] The method is applicable to models containing Swish activation to improve the accuracy of Swish lookup tables, including but not limited to image classification and object detection in computer vision, Transformer models, machine translation and text generation models in natural language processing.

[0033] Therefore, this solution has the following advantages:

[0034] This paper elucidates the principles of Swish's lookup tables and methods to improve their accuracy, aiming to enhance the model's quantization precision, increase its operational efficiency, and facilitate its rapid deployment. Attached Figure Description

[0035] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0036] Figure 1 This is a schematic diagram of the Swish activation function graph.

[0037] Figure 2 This is a flowchart illustrating the method described in this application.

[0038] Figure 3 This is a code example diagram illustrating the search implementation of step S1 of this method.

[0039] Figure 4 This is a schematic diagram of the main code implementation of step S2 in this method. Detailed Implementation

[0040] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the graph of the Swish activation function reveals that when the independent variable x is less than 5, the function value is almost the same, i.e., Swish(x) ≈ Swish(-5). When x < -5, the function value remains constant. To address this characteristic, this invention proposes a method to improve the accuracy of the Swish lookup table. The following is a detailed description of this method. Figure 2 As shown, it includes:

[0042] S1. Lookup table implementation

[0043] 1) Key parameters of the lookup table:

[0044] Number of entries: Number of sampling points on the X-axis;

[0045] Table bit width (LBIT): The bit width of the fixed-point input X;

[0046] Bit width of an entry (TBIT): The fixed-point bit width for storing the sampled values ​​corresponding to the y-axis.

[0047] The process of creating the table:

[0048] Divide x into 32 equal points:

[0049] Find the y-value of the corresponding point;

[0050] Save y when creating the table i and dy;

[0051] 2) Table lookup process:

[0052]

[0053] calculate

[0054] The integer part is i, and the fractional part is ceof;

[0055] 3) The table entries are directly stored and output after quantization:

[0056] y0_uint8 and dy_uint8,

[0057] The input for the table lookup is x_uint8

[0058]

[0059] The input bit width in_bit is divided into table_bit and float_bitwidth.

[0060] Further, find code examples of implementations, such as... Figure 3 As shown, it includes:

[0061] S1.1. Input parameter processing:

[0062] The function `activa_call_linear_tf` receives input data `x_new` and lookup table parameter `lut_param`; it calculates the floating-point bit width `float_bitwidth`, which is the value of `LUT_L_BIT` (input bit width) minus `table_items_bitwidth` (table item bit width);

[0063] Generate a mask_float to extract the decimal part of the input data.

[0064] S1.2. Input Data Preprocessing

[0065] Convert the input data x_new to int32 type and reshape it into a two-dimensional tensor (shape (num, 1), where num is the number of data points);

[0066] The index x_new_indices is calculated by right-shifting x_new by float_bitwidth bits (equivalent to dividing by 2^float_bitwidth).

[0067] S1.3. Lookup Table Queries

[0068] Extract LUT_dy (difference table) and LUT_y0 (base value table) from the lookup table parameter lut_param; use x_new_indices as index to retrieve the corresponding dy and y0 values ​​from the table and convert them to float32 type.

[0069] S1.4. Interpolation Calculation

[0070] Extract the fractional part of the input data x_new using ceof, achieved through bitwise AND operation bitwise_and and mask_float.

[0071] The interpolation result y_new is calculated using the following formula:

[0072] y_new=(ceof×dy+y0×2 float_bitwidth )÷2 float_bitwidth ;

[0073] The final fixed-point conversion is completed through integer division operations.

[0074] S1.5. Return Result Returns the interpolated result y_new, whose shape is consistent with the input data x_new.

[0075] The key points of the above section are:

[0076] Floating-point bit width segmentation: The input bit width is divided into high-order bits (index part) and low-order bits (fractional part) for lookup table indexing and linear interpolation.

[0077] Linear interpolation: Piecewise linear approximation is achieved using the base value y0 and the difference dy, improving the accuracy of the lookup table. Hardware-friendly design: Utilizing bitwise operations and fixed-point arithmetic, it is suitable for hardware-accelerated implementation.

[0078] S2, improves the accuracy of Swish lookup tables.

[0079] To address the characteristics of Swish functions, truncating the input of the Swish function at -5 before calculating the quantization parameters of the Swish layer increases the utilization of the lookup table and effectively improves the accuracy of the Swish layer's lookup table. Specifically, the graph is traversed, and a flag `is_swish_input` is set for the Swish input. When quantizing the parameters, if `is_swish_input` is true, the output of the current layer is truncated at -5.

[0080] Specifically, the Swish input is truncated at -5, and the main code implementation is as follows: Figure 4 As shown, it includes:

[0081] 1) Define an integer variable swish_input_min and initialize it to -5;

[0082] 2) Check if the Boolean variable is_swish_input is true:

[0083] If false, skip all subsequent operations;

[0084] If true, then perform the following operation:

[0085] a. Begin a loop that iterates through each element in the container `quantize_features_data[node_op.name()]`:

[0086] i. Check if the value of the current element is less than swish_input_min(-5);

[0087] ii. If it is less than, then set the value of the current element to swish_input_min(-5);

[0088] iii. If it is not less than, keep the original value unchanged;

[0089] b. The loop continues until all elements in the container have been processed.

[0090] The main function of this code is:

[0091] When is_swish_input is true, all feature data of a specific node (node_op.name()) in quantize_features_data are truncated to a lower bound to ensure that no value is less than -5.

[0092] Compare the cosine similarity of the YOLOv5s model before and after optimization:

[0093] The cosine similarity of the model before optimization was 0.978274.

[0094] The cosine similarity of the optimized model is 0.994623.

[0095] In summary, this invention provides a method to improve the accuracy of Swish lookup tables. It addresses the principles of Swish lookup tables and aims to enhance the quantization accuracy and operational efficiency of the model, thereby facilitating its rapid deployment.

[0096] The technical solution of this invention has a wide range of applications. Models containing Swish activation can use this method to improve the accuracy of Swish lookup tables, such as:

[0097] 1. Computer Vision

[0098] a) Image classification: Swish activation is widely used in the EfficientNet series because its smooth gradient improves the training stability of deep networks and significantly improves the accuracy of ImageNet.

[0099] b. Object detection: Some studies have replaced ReLU with Swish in YOLO variants or Faster R-CNN to alleviate the gradient vanishing problem by utilizing its non-saturation property.

[0100] 2. Natural Language Processing

[0101] a. Transformer model: Swish has replaced ReLU in some studies for use in feedforward network (FFN) layers, improving the ability to model long sequences through smoother activation.

[0102] b. Machine translation and text generation: Experiments show that Swish may alleviate gradient vanishing in LSTM / GRU, but the hyperparameters need to be adjusted according to the specific task.

[0103] This technical solution can be applied to all series of Ingenic chips and similar chips.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for improving the accuracy of Swish lookup tables, characterized in that, The method includes the following steps: S1. Implementation using a lookup table: Determine the main parameters of the lookup table, including the number of entries, the fixed-point width (LBIT) of the input X, and the fixed-point width (TBIT) of the entry storage. Divide the X-axis into 32 sampling points and calculate the Y value corresponding to each sampling point; Store the quantized output values ​​y0_uint8 and dy_uint8; When looking up a table, calculate the coefficient _ceof and the index i, where i is the integer part and _ceof is the decimal part; input x_uint8 and use the formula... Quantification processing is performed; S2. Improve the accuracy of Swish lookup tables: To address the input characteristics of the Swish function, the input is truncated at -5 before statistical quantization. Traverse the computation graph and set the flag is_swish_input for Swish's input; When is_swish_input is true, the output of the current layer is truncated at -5 to improve the utilization and accuracy of the lookup table.

2. The method for improving the accuracy of Swish lookup tables according to claim 1, characterized in that, Step S1 further includes: S1.

1. Input parameter processing: Receive input data x_new and lookup table parameter lut_param, calculate floating-point bit width float_bitwidth, and generate mask mask_float for extracting the decimal part of the input data; S1.

2. Input data preprocessing: Convert the input data x_new to int32 type and reshape it into a two-dimensional tensor, and calculate the index x_new_indices through a right shift operation; S1.

3. Lookup table query: Extract the difference table LUT_dy and the base value table LUT_y0 from the lookup table parameter lut_param, use x_new_indices as the index to get the corresponding dy and y0 values, and convert them to float32 type; S1.

4. Interpolation calculation: Extract the decimal part ceof of the input data x_new, calculate the interpolation result y_new using the linear interpolation formula, and perform fixed-point processing; S1.

5. Return Result: Output the interpolated result y_new, whose shape is consistent with the input data x_new.

3. The method for improving the accuracy of the Swish lookup table according to claim 1, characterized in that, The lookup table has 32 entries.

4. The method for improving the accuracy of the Swish lookup table according to claim 1, characterized in that, In the table lookup process, the formula for calculating index i is: Where table_num represents the number of table entries.

5. The method for improving the accuracy of the Swish lookup table according to claim 1, characterized in that, In the quantization process of the input x_uint8, the input bit width in_bit is divided into two parts: table_bit and float_bitwidth.

6. The method for improving the accuracy of the Swish lookup table according to claim 1, characterized in that, The input truncation operation of the Swish function is used to improve the utilization and accuracy of the lookup table.

7. The method for improving the accuracy of the Swish lookup table according to claim 1, characterized in that, The method is applicable to models containing Swish activation to improve the accuracy of Swish lookup tables, including but not limited to image classification and object detection in computer vision, Transformer models, machine translation and text generation models in natural language processing.

Images