Deep neural network quantization method and system based on beta divergence and soft approximation mapping

Abstract

The invention discloses a deep neural network quantization method and system based on beta divergence and soft approximation mapping. According to the method, dynamic distribution statistics is carried out on input data of each batch of a network to obtain data distribution information, AIC is introduced to serve as an evaluation index of distribution goodness of fit, optimal fitting distribution is dynamically selected from multiple distribution models such as normal distribution models, Laplace distribution models and generalized Gaussian distribution models, and the optimal quantization interval is accurately determined; and then constructing a derivable soft rounding approximation function, designing a dynamic slope adjustment mechanism, and adaptively adjusting a slope parameter of a mapping function according to the variance activated and output of each layer, so that the mapping function realizes dynamic balance of quantization and precision in different distribution states. After network training is completed, original floating point number weights are recorded, symmetric quantization is performed to obtain 8-bit integer weights with sign bits, the difference of weight distribution before and after quantization is measured by using improved beta divergence, and quantization weights are guided to approach original floating point weight distribution in a retraining stage, so that quantization errors are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning model optimization, and specifically relates to a deep neural network quantization method and system based on beta divergence and soft approximation mapping. Background Art

[0002] With the rapid development of artificial intelligence (AI) in various fields, deep neural networks have been widely applied to tasks such as image recognition, speech processing, and natural language understanding, achieving performance far exceeding that of traditional methods on multiple benchmarks. However, high-performance deep models often come with a large number of parameters and high computational cost, making them difficult to deploy directly on resource-constrained edge devices. For example, ResNet-50 requires approximately 4GFLOPs of computation during inference and occupies over 90MB of storage space. This makes it difficult for embedded devices with only a few hundred MHz CPU and less than 1MB of SRAM to perform inference on various AI tasks. Quantization, a key method for compressing and accelerating neural networks, significantly reduces storage and computational overhead by compressing model parameters and activation maps from floating-point representations to fixed-point integers, thereby improving model efficiency on edge devices. Currently, the main research focus of quantization is on balancing network computational complexity and accuracy. The goal is to maintain network output accuracy as much as possible while significantly reducing network computational complexity. Therefore, deep neural network quantization methods are needed to achieve this balance between network computational complexity and accuracy. Summary of the Invention

[0003] This invention provides a deep neural network quantization method and system, addressing the technical problem that traditional neural network quantization methods often use static quantization intervals and non-differentiable rounding functions, ignoring the dynamic distribution of input data, resulting in large quantization errors and reduced model accuracy. Furthermore, most current quantization methods employ simple distribution matching strategies, making it difficult to precisely control quantization errors. Deployment on edge devices still presents challenges such as high model parameter count, high inference latency, and limited computing resources.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A deep neural network quantization method based on beta divergence and soft approximation mapping, comprising:

[0006] Step 1: Data Preprocessing

[0007] S1: When training a deep neural network, the distribution characteristics of each batch of input data are statistically analyzed to obtain the data distribution characteristics corresponding to each batch of input data. The data distribution characteristics include the mean, standard deviation, skewness, and kurtosis of the data;

[0008] Step 2: Activate Quantization

[0009] S2: Determine the quantization interval of each batch of input data based on the data distribution characteristics corresponding to each batch of input data;

[0010] S3: During the forward pass of the deep neural network, when an intermediate layer in the deep neural network uses an activation function, a soft rounding approximation function with a dynamic slope adjustment mechanism is used to quantize the output of the activation function of the intermediate layer according to the quantization interval [a, b] generated by S2, so that the tensor in the network transmission is converted from a floating point number to a signed integer of a preset bit width to serve as the input of the next layer of the deep neural network, where a and b represent the minimum and maximum values ​​of the quantization interval respectively;

[0011] Step 3: Weight Quantization

[0012] S4: After performing activation quantization according to step 2 during the deep neural network training process and completing the deep neural network training, obtain the weights of the deep neural network, that is, the original floating-point weights;

[0013] S5: symmetrically quantize the original floating-point weights, convert the original floating-point weights into signed integers of a preset bit width, and generate integer weights;

[0014] Step 4: Retrain

[0015] S6: Retrain the deep neural network using the original floating-point weights and the quantized integer weights. During the forward pass of the deep neural network, the integer weights are dequantized to floating-point weights.

[0016] S7: Perform histogram statistics on the floating-point weights generated by dequantization and the original floating-point weights respectively;

[0017] S8: Based on the histogram statistics, the beta divergence is calculated, and based on the result of the beta divergence calculation and the mean square error loss, the loss function of the retraining process is obtained, and the deep neural network is retrained until the loss function converges.

[0018] Furthermore, the preset bit width is 8 bits.

[0019] Furthermore, S2 specifically includes:

[0020] When the kurtosis is greater than the preset kurtosis threshold, the quantization interval is calculated using the Laplace distribution;

[0021] When the kurtosis is less than the preset kurtosis threshold and the absolute value of the skewness is less than the preset skewness threshold, the quantization interval is calculated using the normal distribution;

[0022] When the kurtosis is less than the preset kurtosis threshold and the absolute value of the skewness is not less than the preset skewness threshold, the generalized Gaussian distribution is used to calculate the quantization interval

[0023] Furthermore, S2 specifically includes:

[0024] For each batch of input data, perform the following steps:

[0025] S21: Based on the data distribution characteristics corresponding to the batch input data, a candidate distribution model is used to

[0026] Fit the input data;

[0027] S22: Calculate the AIC value corresponding to each candidate distribution model;

[0028] S23: Using the candidate distribution model with the smallest AIC value as the distribution model of the batch input data;

[0029] S24: Determine a quantization interval of the batch input data according to a distribution model of the batch input data.

[0030] Furthermore, candidate distribution models include normal distribution, Laplace distribution, and generalized Gaussian distribution. S24 specifically includes:

[0031] When the distribution model of the batch input data is a normal distribution, the quantization interval is: [a, b] = [μ - kσ, μ + kσ], where [a, b] represents the quantization interval, μ represents the mean of the batch input data, σ represents the standard deviation of the batch input data, and k is a preset first constant used to control the range of input data covered by the quantization interval;

[0032] When the distribution model of the batch input data is Laplace distribution, the quantization interval is: Where b0 represents the scale parameter in the Laplace distribution, ε1 represents a preset second constant, which is used to control the range of input data covered by the quantization interval;

[0033] When the distribution model of the batch input data is a generalized Gaussian distribution, the quantization interval is: [a, b] = [μ-T, μ+T], where α and β0 represent the scale parameter and shape parameter in the generalized Gaussian distribution respectively, and ε2 represents a preset third constant used to control the range of input data covered by the quantization interval.

[0034] Furthermore, the soft rounding approximation function with a dynamic slope adjustment mechanism is:

[0035]

[0036] in,

[0037] q(x′) represents a soft rounding approximation function with a dynamic slope adjustment mechanism that performs soft approximation mapping on x′;

[0038] clip(·) represents the clipping function;

[0039] represents the dynamic slope, Represents the variance of the output value x′ of the activation function of the current intermediate layer, and ε0 is a constant to prevent the denominator from being zero and causing instability;

[0040] Indicates rounding down;

[0041] Represents the quantization factor, that is, the floating-point interval corresponding to each integer unit;

[0042] M and -N represent the maximum and minimum values ​​of the signed integer of the preset bit width, i.e. the clipping boundary values;

[0043] τ represents the preset center offset of the Sigmoid function.

[0044] Furthermore, the symmetric quantization in S5 is expressed as:

[0045]

[0046] in,

[0047] w i represents the i-th original floating-point weight, is the quantized integer weight corresponding to the i-th original floating-point weight, i∈{1,2,...,N0}, N0 represents the total number of original floating-point weights;

[0048] round(·) represents a hard rounding function.

[0049] Furthermore, in S6, the integer weights are dequantized into floating-point weights using the formula:

[0050]

[0051] in, Indicates that The floating-point weights generated by dequantization.

[0052] Furthermore, the beta divergence calculation formula in S8 is:

[0053]

[0054] in,

[0055] Indicates the beta divergence calculation of P and Q;

[0056] P={p1,p2,...,p K}, p j Indicates the probability that the original floating-point weight falls into the jth bin in the histogram when the original floating-point weight is subjected to histogram statistics, j∈{1,2,...,K}, where K represents the total number of bins in the histogram;

[0057] Q={q1,q2,...,q K},q j Indicates the probability that the floating-point weight generated by dequantization falls into the jth bin in the histogram when the histogram statistics are performed on the floating-point weight generated by dequantization;

[0058] β represents a hyperparameter that controls the shape of the divergence.

[0059] The present invention also provides a deep neural network quantization system based on beta divergence and soft approximation mapping, the system comprising:

[0060] a memory configured to store a computer program;

[0061] A processor is configured to execute the computer program to implement the deep neural network quantization method based on beta divergence and soft approximation mapping as described above.

[0062] The beneficial effects of the present invention are:

[0063] The present invention obtains data distribution information by performing dynamic distribution statistics (including mean, standard deviation, skewness, and kurtosis) on the data input to each batch (i.e., batch) of the network, determines the precise quantization interval in combination with the data distribution information, designs a differentiable soft rounding approximation function, and performs activation function quantization during the forward transmission process of each layer of the network. After the network training is completed, the original floating-point weights are recorded, and the weights are quantized to obtain signed integer weights of a preset bit width. The improved beta divergence is used to measure the difference in weight distribution before and after quantization for retraining, guiding the quantized weights to gradually approach the original weight distribution, thereby reducing the error caused by weight quantization. In the overall reasoning process, weights and activations are both represented by integers with the same bit width (e.g., 8 bits), reducing the network floating-point operations (GFLOPS) while ensuring the stability of model accuracy, thereby achieving efficient deployment and reasoning on resource-constrained edge devices. This invention mainly addresses the problem that resource-constrained edge devices are unable to perform large-scale model task reasoning. It designs a soft quantization strategy that activates distribution perception and β divergence guidance, reduces the GFLOPS of the network model, and enables resource-constrained edge devices to perform model task reasoning, effectively improving the stability and reasoning performance of the model under low precision, and providing a cost-effective deep neural network optimization solution for edge computing scenarios.

[0064] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0066] Figure 1 This is a schematic diagram of a network framework of a deep neural network quantization method based on beta divergence and soft approximate mapping provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0068] like Figure 1The network framework diagram shown in FIG. 1 shows an embodiment of the present invention providing a deep neural network quantization method based on β divergence and soft approximation mapping, which includes: (1) data preprocessing; (2) activation quantization based on soft approximation mapping; (3) weight quantization based on β divergence; and (4) retraining after weight quantization. Each step is described in detail below.

[0069] Step 1: Data Preprocessing

[0070] S1: When training a deep neural network, the distribution characteristics of each batch of input data are statistically analyzed to obtain the data distribution characteristics corresponding to each batch of input data. The data distribution characteristics include the mean, standard deviation, skewness, and kurtosis of the data.

[0071] Assume that the input data of each batch of the network is Among them, N b is the number of data elements in each batch, then the mean μ can be expressed by formula (1) as:

[0072]

[0073] The data standard deviation σ can be expressed by formula (2):

[0074]

[0075] The data standard deviation is used to measure the degree of data dispersion. The smaller the standard deviation, the more concentrated the data is, and it is easier to be accurately expressed by a small range of integer values ​​in quantization; the larger the standard deviation, the more discrete the data is, and a larger quantization range is required in subsequent quantization.

[0076] The skewness γ1 can be expressed by formula (3):

[0077]

[0078] Data skewness is a statistic used to measure the degree of asymmetry in a data set. If the skewness is equal to 0, it means that the distribution is symmetrical and approximately normal, which is suitable for symmetric quantification in subsequent quantification. If the skewness is greater than 0, most of the data values ​​are concentrated on the left. If the skewness is less than 0, most of the data values ​​are concentrated on the right. When the skewness is not equal to 0, it is more suitable for asymmetric quantification.

[0079] The kurtosis γ2 can be expressed by formula (4):

[0080]

[0081] Data kurtosis is used to measure whether there are extreme cases in the data distribution. If the data kurtosis is greater than the preset kurtosis threshold (for example, 3), it means that the data is highly concentrated, sharp in the middle, but has many outliers at the tail, which often requires greater clipping during quantification.

[0082] Step 2: Activate Quantization

[0083] S2: Determine the quantization interval of each batch of input data based on the data distribution characteristics corresponding to each batch of input data;

[0084] S3: During the forward pass of the deep neural network, when the intermediate layer in the deep neural network uses an activation function, the activation value of the intermediate layer is quantized using a soft rounding approximation function with a dynamic slope adjustment mechanism according to the quantization interval [a, b] generated by S2, so that the tensor in the network transmission is converted from a floating point number to a signed integer of a preset bit width to serve as the input of the next layer of the deep neural network, where a and b represent the minimum and maximum values ​​of the quantization interval, respectively.

[0085] In some embodiments, the quantization interval may be determined directly based on the data distribution characteristics of each batch input. For example, the quantization interval may be determined based on the kurtosis and skewness values ​​of the data.

[0086] When the kurtosis is greater than a preset kurtosis threshold (e.g., 3), the quantization interval is calculated using Laplace distribution;

[0087] When the kurtosis is less than a preset kurtosis threshold (e.g., 3) and the absolute value of the skewness is less than a preset skewness threshold, the normal distribution is used to calculate the quantization interval. The preset skewness threshold can be any appropriate value, for example, it can be designed to be 0.01, 0.001, etc. In this case, the skewness can be considered to be approximately 0, which means that the data distribution is symmetrical.

[0088] When the kurtosis is less than the preset kurtosis threshold and the absolute value of the skewness is not less than the preset skewness threshold (ie, the skewness is not a number approximately equal to 0, indicating that the distribution of the data is asymmetric), the generalized Gaussian distribution is used to calculate the quantization interval.

[0089] In some embodiments, in order to select a suitable quantitative mapping model based on the distribution characteristics of the input data, a mixed distribution fitting mechanism can be introduced to model the input data using multiple candidate distribution models, and the fitting effect of each model can be evaluated in combination with AIC (i.e., Akaike Information Criterion). AIC can be expressed as

[0090] AIC=2k′-2ln(L), (5)

[0091] Here, k′ represents the number of model parameters, and L represents the maximum likelihood function value of the model. This AIC metric automatically selects the optimal probability distribution while balancing fitting accuracy and model complexity. This criterion-based model selection method dynamically adapts to the distribution differences of data at different layers, thereby more appropriately determining the quantization interval.

[0092] S2 may specifically include the following steps:

[0093] For each batch of input data, perform the following steps:

[0094] S21: Based on the data distribution characteristics corresponding to the batch input data, a candidate distribution model is used to

[0095] Fit the input data;

[0096] S22: Calculate the AIC value corresponding to each candidate distribution model;

[0097] S23: Using the candidate distribution model with the smallest AIC value as the distribution model of the batch input data;

[0098] S24: Determine a quantization interval of the batch input data according to a distribution model of the batch input data.

[0099] Among them, the candidate distribution models include normal distribution, Laplace distribution and generalized Gaussian distribution. The probability density functions of these distribution models are expressed by formula (6)-formula (8) respectively as follows:

[0100]

[0101] Among them, p1(x) represents the normal distribution probability density function when the data x obeys the normal distribution, p2(x) represents the Laplace distribution probability density function, and b0 represents the scale parameter in the Laplace distribution, which determines the distribution width of the data and has the following relationship with the standard deviation of the data: p3(x) is the probability density function of the generalized Gaussian distribution, Γ(·) is the gamma function, and α and β0 represent the scale parameter and shape parameter in the generalized Gaussian distribution, respectively.

[0102] The log-likelihood values ​​of the above distribution models can be expressed by formula (9)-formula (11) respectively as follows:

[0103]

[0104] Among them, L normal , L laplace and L ggdThe likelihood functions for the normal, Laplace, and generalized Gaussian distributions are represented by the logarithm (ln), respectively. Using the above formula, we can obtain the AIC values ​​of the candidate distribution models. The distribution model with the smallest AIC value can be selected as the distribution model for the data set.

[0105] When the input data distribution model is normal, the quantization interval is: [a, b] = [μ - kσ, μ + kσ], where [a, b] represents the quantization interval, μ represents the mean of the batch input data, σ represents the standard deviation of the batch input data, and k is a preset first constant used to control the range of input data covered by the quantization interval. As an example, k can be 2, 3, or any other suitable value. When it is 2, it can cover approximately 95.4% of the data; when it is 3, it can cover approximately 99.7% of the data.

[0106] When the distribution model of the input data is Laplace distribution, since Laplace distribution has an explicit form, the truncation range can be derived according to the coverage probability P0.

[0107] Assuming P0 = 1-2ε1, the quantization interval is: ε1 represents a preset second constant, which is used to control the range of input data covered by the quantization interval. Generally, when ε1 is 0.01, it can cover 98% of the data range.

[0108] When the distribution model of the input data is a generalized Gaussian distribution, the quantization interval is: [a, b] = [μ-T, μ+T], where ε2 represents the preset third constant, which is used to control the input data range covered by the quantization interval. ε2 is usually set to 10 -3 .

[0109] After determining the quantization interval, when the activation function is used in the intermediate layer of the network forward pass, a differentiable soft rounding approximation function can be used to quantize the activation function. In some embodiments, to control the smoothness of the mapping, a dynamic slope adjustment mechanism can be used to adjust the slope parameter k0 of the sigmoid function in the soft rounding function, where the value of k0 can be adaptively associated with the variance of the activation value of the activation output of the current layer (i.e., the current intermediate layer).

[0110] Specifically, the soft rounding approximation function with a dynamic slope adjustment mechanism can be expressed by formula (12) as:

[0111]

[0112] in,

[0113] q(x′) represents a soft rounding approximation function with a dynamic slope adjustment mechanism that performs a soft approximation mapping on the output value x′ (i.e., the activation value) of the activation function;

[0114] clip(·) represents a clipping function, which is used to limit the range of values; Indicates rounding down; Represents the quantization factor, that is, the floating-point interval corresponding to each integer unit; M and -N respectively represent the maximum and minimum values ​​of the signed integer of the preset bit width, that is, the clipping boundary value. When using 8-bit signed integers, M and N are 127; Indicates the normalization of x; τ represents the preset center offset of the Sigmoid function, which can usually be taken as 0.5 according to previous studies;

[0115] represents the dynamic slope, represents the variance of the output value x′ of the activation function of the current intermediate layer. ε0 is a constant to prevent the denominator from being 0 and causing instability. For example, it can be taken as 10 -6 When the activation distribution of a certain layer is more concentrated (i.e., the variance is small), the mechanism will automatically increase the k0 value to make the rising range of the Sigmoid function steeper and the mapping closer to hard quantization, which is conducive to preserving numerical resolution; when the activation distribution is more dispersed, the mechanism will reduce the k0 value to maintain a smoother approximation to maintain gradient transferability.

[0116] Therefore, through the dynamic slope adjustment mechanism, the improved soft rounding approximation function can adopt a slope that matches the distribution characteristics of each layer, thereby improving the overall activation quantization accuracy and retaining the differentiability to support subsequent backpropagation training.

[0117] Step 3: Weight Quantization

[0118] S4: After performing activation quantization according to step 2 and completing the deep neural network training during the deep neural network training process, obtain the weights of the deep neural network, that is, the original floating-point weights, and record them;

[0119] S5: symmetrically quantize the original floating-point weight (i.e., by mapping the zero point to 0), convert the original floating-point weight to a signed integer of a preset bit width, generate an integer weight, and record it;

[0120] Symmetric quantization can be expressed by formula (13):

[0121]

[0122] Among them, w i represents the i-th original floating-point weight, is the quantized integer weight corresponding to the i-th original floating-point weight, i∈{1,2,…,N0}, N0 represents the total number of original floating-point weights; round(·) represents the hard rounding function.

[0123] Step 4: Retrain

[0124] S6: Retrain the deep neural network using the original floating-point weights and the quantized integer weights. During the forward pass of the deep neural network, the integer weights are dequantized to floating-point weights.

[0125] The dequantization of integer weights into floating-point weights is expressed by formula (14):

[0126]

[0127] in, Indicates that The floating-point weights generated by dequantization.

[0128] S7: Perform histogram statistics on the floating-point weights generated by dequantization and the original floating-point weights respectively;

[0129] S8: Based on the histogram statistics, the beta divergence is calculated, and based on the result of the beta divergence calculation and the mean square error loss MSE, the loss function of the retraining process (i.e., the total loss function) is obtained, and the deep neural network is retrained until the loss function converges.

[0130] In some embodiments, the process of calculating the beta divergence in S8 is expressed by formula (14):

[0131]

[0132] in,

[0133] Indicates the beta divergence calculation of P and Q;

[0134] P={p1,p2,...,p K}, p j Indicates the probability that the original floating-point weight falls into the jth bin in the histogram when the original floating-point weight is subjected to histogram statistics (that is, the number of weights that fall into this bin divided by the total number of weights), j∈{1,2,...,K}, K represents the total number of bins in the histogram;

[0135] Q={q1,q2,...,q K},q j Indicates the probability that the floating-point weight generated by dequantization falls into the jth bin in the histogram when the histogram statistics are performed on the floating-point weight generated by dequantization;

[0136] β is a hyperparameter that controls the shape of the divergence and can be set to 2.0.

[0137] The beta divergence mentioned above is used as the primary loss function to measure the difference between the original floating-point weights and the dequantized floating-point weights. Training is continued until convergence. During retraining, the loss will continue to decrease until convergence, indicating that the quantized weight distribution is infinitely close to the original floating-point weight distribution, greatly reducing the quantization error caused by weight quantization, thereby significantly reducing the network computation load and minimizing network accuracy loss.

[0138] The specific process of retraining is:

[0139] After completing the training of the original network and obtaining the floating-point weights (i.e., the original floating-point weights), the original floating-point weights are first symmetrically quantized to obtain quantized weights in integer form with a preset bit width (e.g., 8-bit). In order to reduce the loss of model accuracy introduced by quantization, the quantized weights are subsequently optimized through a retraining mechanism. At the beginning of each round of iteration, the quantized weights are first dequantized into floating-point form, and histograms are constructed with the original floating-point weights to obtain the corresponding normalized probability distributions P and Q. In the forward propagation process of retraining, the β divergence is used as the optimization target to measure the difference between the current dequantized floating-point weight distribution and the original (i.e., original floating-point weight) distribution. The conventional mean square error function is combined to form a joint loss function, and gradient backpropagation is performed to optimize the network parameters. That is, the floating-point weights in the network are adjusted, and then the adjusted floating-point weights (that is, the original floating-point weights corresponding to this iteration) are quantized to generate the quantized weights of this iteration, and then dequantized to generate the dequantized floating-point weights of this iteration. Then, a histogram is constructed for the original floating-point weights corresponding to this iteration and the floating-point weights generated by dequantization to obtain the corresponding normalized probability distributions P and Q (that is, each round of iteration will generate corresponding P and Q), and the β divergence is calculated according to formula (14) and the mean square error is determined according to formula (15), thereby determining the total loss function. If the total loss function does not converge, the floating-point weights in the network are continued to be adjusted to enter the next round of iteration, and the above process is repeated until the total loss function converges.

[0140] The mean square error (MSE) function is expressed by formula (15):

[0141]

[0142] Then the total loss function (i.e., joint loss function) after combining β divergence and MSE is expressed as follows using formula (16):

[0143]

[0144] in, and y m They represent the two probability distributions input into the MSE function, namely, p in formula (14) j and q j λ1 and λ2 are weight coefficients used to balance task loss and distribution preservation. λ1 is typically set to 1.0, and λ2 is typically set to 0.01 to 0.1. Network training terminates when the total loss function converges, completing the final quantized model generation. The entire retraining process, by fitting the weight distribution, guides the floating-point weights generated by dequantization to approximate the original distribution with minimal divergence. This effectively controls model complexity and hardware execution overhead while ensuring network inference accuracy.

[0145] The present invention also provides a deep neural network quantization system based on beta divergence and soft approximation mapping, the system comprising:

[0146] a memory configured to store a computer program;

[0147] A processor is configured to execute the computer program to implement the deep neural network quantization method based on beta divergence and soft approximation mapping as described above.

[0148] In summary, the embodiments of the present invention provide a deep neural network quantization method and system based on beta divergence and soft approximation mapping. The present invention performs dynamic distribution statistics on each batch of network input data to obtain data distribution information, introduces AIC as an evaluation indicator of distribution fit, dynamically selects the best fitting distribution from multiple distribution models such as normal, Laplace, and generalized Gaussian, and accurately determines the optimal quantization interval. A differentiable soft rounding approximation function is constructed, and a dynamic slope adjustment mechanism is designed to adaptively adjust the mapping function slope parameter according to the variance of the activation output of each layer, so that it can achieve a dynamic balance between quantization and accuracy under different distribution states. After the network training is completed, the original floating-point weights are recorded and symmetrically quantized to obtain 8-bit signed integer weights. The beta divergence is used to measure the difference in weight distribution before and after quantization. During the retraining stage, the quantized weights are guided to approach the original floating-point weight distribution, thereby reducing the quantization error. In the overall inference process, both weights and activations are represented by 8-bit integers, which significantly reduces GFLOPS while ensuring stable model accuracy, realizing low-power and high-efficiency model inference for edge devices. To address the problem that resource-constrained edge devices are unable to perform large-scale model tasks, the present invention designs a soft quantization strategy with activation distribution awareness, adaptive mapping slope adjustment, and a weight fitting mechanism based on β-divergence. This provides a deep neural network optimization solution for edge computing scenarios that is cost-effective and has high inference performance under low precision.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A deep neural network quantization method based on β divergence and soft approximation mapping, characterized in that: include: Step 1: Data Preprocessing S1: When training a deep neural network, the distribution characteristics of each batch of input data are statistically analyzed to obtain the data distribution characteristics corresponding to each batch of input data. The data distribution characteristics include the mean, standard deviation, skewness, and kurtosis of the data; Step 2: Activate Quantization S2: Determine the quantization interval of each batch of input data based on the data distribution characteristics corresponding to each batch of input data; S3: During the forward pass of the deep neural network, when an intermediate layer in the deep neural network uses an activation function, a soft rounding approximation function with a dynamic slope adjustment mechanism is used to quantize the output of the activation function of the intermediate layer according to the quantization interval [a, b] generated by S2, so that the tensor in the network transmission is converted from a floating point number to a signed integer of a preset bit width to serve as the input of the next layer of the deep neural network, where a and b represent the minimum and maximum values ​​of the quantization interval respectively; Step 3: Weight Quantization S4: After performing activation quantization according to step 2 during the deep neural network training process and completing the deep neural network training, obtain the weights of the deep neural network, that is, the original floating-point weights; S5: symmetrically quantize the original floating-point weights, convert the original floating-point weights into signed integers of a preset bit width, and generate integer weights; Step 4: Retrain S6: Retrain the deep neural network using the original floating-point weights and the quantized integer weights. During the forward pass of the deep neural network, the integer weights are dequantized to floating-point weights. S7: Perform histogram statistics on the floating-point weights generated by dequantization and the original floating-point weights respectively; S8: Based on the histogram statistics, the beta divergence is calculated, and based on the result of the beta divergence calculation and the mean square error loss, the loss function of the retraining process is obtained, and the deep neural network is retrained until the loss function converges.

2. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 1, characterized in that: The preset bit width is 8 bits.

3. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 1, characterized in that: S2 specifically includes: When the kurtosis is greater than the preset kurtosis threshold, the quantization interval is calculated using the Laplace distribution; When the kurtosis is less than the preset kurtosis threshold and the absolute value of the skewness is less than the preset skewness threshold, the quantization interval is calculated using the normal distribution; When the kurtosis is less than the preset kurtosis threshold and the absolute value of the skewness is not less than the preset skewness threshold, the generalized Gaussian distribution is used to calculate the quantization interval.

4. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 1, characterized in that: S2 specifically includes: For each batch of input data, perform the following steps: S21: fitting the batch input data using a candidate distribution model according to data distribution characteristics corresponding to the batch input data; S22: Calculate the AIC value corresponding to each candidate distribution model; S23: Using the candidate distribution model with the smallest AIC value as the distribution model of the batch input data; S24: Determine a quantization interval of the batch input data according to a distribution model of the batch input data.

5. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 4, characterized in that: Candidate distribution models include normal distribution, Laplace distribution, and generalized Gaussian distribution. S24 specifically includes: When the distribution model of the batch input data is a normal distribution, the quantization interval is: [a, b] = [μ - kσ, μ + kσ], where [a, b] represents the quantization interval, μ represents the mean of the batch input data, σ represents the standard deviation of the batch input data, and k is a preset first constant used to control the range of input data covered by the quantization interval; When the distribution model of the batch input data is Laplace distribution, the quantization interval is: Where b0 represents the scale parameter in the Laplace distribution, ε1 represents a preset second constant, which is used to control the range of input data covered by the quantization interval; When the distribution model of the batch input data is a generalized Gaussian distribution, the quantization interval is: [a, b] = [μ-T, μ+T], where α and β0 represent the scale parameter and shape parameter in the generalized Gaussian distribution respectively, and ε2 represents a preset third constant used to control the range of input data covered by the quantization interval.

6. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 1, characterized in that: The soft rounding approximation function with dynamic slope adjustment mechanism is: in, q(x′) represents a soft rounding approximation function with a dynamic slope adjustment mechanism that performs soft approximation mapping on x′; clip(·) represents the clipping function; represents the dynamic slope, Represents the variance of the output value x′ of the activation function, and ε0 is a constant to prevent the denominator from being zero and causing instability; Indicates rounding down; Represents the quantization factor, that is, the floating-point interval corresponding to each integer unit; M and -N represent the maximum and minimum values ​​of the signed integer of the preset bit width, i.e. the clipping boundary values; τ represents the preset center offset of the Sigmoid function.

7. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 6, characterized in that: The symmetric quantization in S5 is expressed as: in, w i represents the i-th original floating-point weight, is the quantized integer weight corresponding to the i-th original floating-point weight, i∈{1,2,...,N0}, N0 represents the total number of original floating-point weights; round(·) represents a hard rounding function.

8. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 7, characterized in that: In S6, the dequantization of integer weights into floating-point weights is expressed as follows: in, Indicates that The floating-point weights generated by dequantization.

9. The deep neural network quantization method based on beta divergence and soft approximation mapping according to claim 8, characterized in that: The beta divergence calculation formula in S8 is: in, Indicates the beta divergence calculation of P and Q; P={p1,p2,...,p K }, p j Indicates the probability that the original floating-point weight falls into the jth bin in the histogram when the original floating-point weight is subjected to histogram statistics, j∈{1,2,...,K}, where K represents the total number of bins in the histogram; Q={q1,q2,...,q K },q j Indicates the probability that the floating-point weight generated by dequantization falls into the jth bin in the histogram when the histogram statistics are performed on the floating-point weight generated by dequantization; β represents a hyperparameter that controls the shape of the divergence.

10. A deep neural network quantization system based on beta divergence and soft approximation mapping, characterized in that: include: a memory configured to store a computer program; A processor configured to execute the computer program to implement the deep neural network quantization method based on beta divergence and soft approximation mapping according to any one of claims 1 to 9.

Images