A deep neural network adaptive quantization method and system based on joint optimization

Abstract

The application provides a deep neural network adaptive quantization method and system based on joint optimization, which comprises the following steps: joint optimization of division parameters and neural network parameters, so that data division of the deep neural network and neural network parameter training are performed simultaneously; saving the division parameters after training; quantizing the activation or weight of the execution model by using the division parameters to obtain a final model; and executing the corresponding task by using the final model. The application can better fit the data distribution of the activation and weight of the deep neural network, quantize the activation value and weight of all linear layers in the deep neural network into low bits, thereby reducing the quantization error, and in the use stage of the model, the application can reduce the calculation amount of the model, accelerate the prediction speed of image classification, reduce the energy consumption generated in classification, and reduce the space size used for model storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of quantization acceleration of deep neural networks and the technical field of picture classification, and particularly relates to a deep neural network adaptive quantization method and system based on joint optimization. BACKGROUND

[0002] In the field of deep learning, in order to reduce the training complexity and speed up the training, a quantization-aware training is used for deep neural networks, that is, the activation values and weights of linear layers (including convolutional layers and fully connected layers) in the deep neural network are quantized to low-bit-width (such as 16-bit, 8-bit) fixed-point numbers or floating-point numbers during the training process.

[0003] The existing quantization-aware training of deep neural networks uses uniform quantization or non-uniform quantization of fixed format (such as floating-point number FP8). However, in actual use, the data distribution of different deep neural networks is different. The existing quantization methods use the same way to allocate quantization intervals, lack of adaptability to different data, and thus cannot effectively allocate quantization intervals when using low-bit-width, so as to obtain good quantization effect. SUMMARY

[0004] The purpose of the present application is to solve the problem that the prior art is difficult to allocate appropriate quantization intervals for the activation and weight of the deep neural network, and to ensure that the activation and weight are quantized to low-bit-width data and maintain the network accuracy effect. In actual application, in order to maintain the network accuracy effect, it is necessary to reduce the quantization error, that is, to reduce the MSE mean square error of the data after quantization and the data before quantization. Based on this purpose, the present application proposes an adaptive quantization-aware training method based on joint optimization.

[0005] The present application proposes an adaptive quantization method for deep neural networks based on joint optimization, which comprises:

[0006] Step 1, obtaining a deep neural network model to be trained as a target model, and counting the maximum absolute value Z of the activation or weight in all linear layers of the target model; according to the maximum absolute value Z, initializing the segment division parameters of the activation or weight setting of each linear layer

[0007] Step 2, when the target model is trained for the t-th time, the segment division parameter value of the activation or weight of the i-th linear layer is with The activation or weight of the i-th linear layer is divided into two parts, and The region is taken as the center segment, and the regions on both sides are and as a surrounding segment; quantizing activations or weights in the target model to low precision, and a quantization interval of activations or weights located in the center segment is smaller than a quantization interval of activations or weights of the surrounding segment during the quantization;

[0008] Step 3, during the forward propagation process of the t-th training iteration of the target model, simultaneously calculating a full-precision forward propagation result and a quantized low-precision forward propagation result of the activations or weights; obtaining quantization errors of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, summing the quantization errors of each layer of the target model, adding a preset coefficient to the loss function of the training result of the target model to obtain a final loss function;

[0009] Step 4, determining whether the final loss function is less than a preset value or t reaches a preset iteration number, if yes, saving the current target model as an execution model, and saving the current segmentation division parameter using the current segmentation division parameter quantizing the activations or weights of the execution model to obtain a final model, and using the final model to perform a corresponding task; otherwise, increasing t by 1, and executing the step 2 again.

[0010] The deep neural network adaptive quantization method based on joint optimization, wherein the target model is an image classification deep neural network model, a target picture to be classified is input into the final model, and the final model performs an image classification task to obtain a category to which the target picture belongs.

[0011] The deep neural network adaptive quantization method based on joint optimization, wherein

[0012] The quantization interval of the activations or weights located in the center segment is:

[0013]

[0014] The quantization interval of the activations or weights located in the surrounding segment is:

[0015]

[0016] The deep neural network adaptive quantization method based on joint optimization, wherein the activations or weights located in the center segment are quantized to:

[0017]

[0018] The activations or weights located in the surrounding segment are quantized to:

[0019]

[0020] wherein It is the full-precision activation or weight value of the i-th linear layer in the t-th iteration. It is the low-precision fixed-point number of the i-th linear layer after quantization;

[0021] The quantization error of each linear layer of the target model Summing, multiplying by a preset coefficient γ, and then applying the result to the loss function of the target model's training outcome. The summation yields the final loss function; the backpropagation process calculates the gradients of the deep neural network parameters and simultaneously calculates the segmentation parameters. The gradient; the segmentation parameters are calculated after backpropagation. The gradient is After the t-th iteration, the segmentation parameters are... The update is performed using the following formula:

[0022]

[0023] Where η is the learning rate, and then... Calculate the (t+1)th iteration.

[0024] This invention also proposes a deep neural network adaptive quantization system based on joint optimization, which includes:

[0025] The initialization module is used to obtain the deep neural network model to be trained as the target model, and to calculate the maximum absolute value Z of the activations or weights in all linear layers of the target model; based on this maximum absolute value Z, the segmentation parameters for the activations or weights of each linear layer are initialized.

[0026] The iterative training module specifies that, during the t-th training iteration of the target model, the piecewise partitioning parameter value for the activation or weights of the i-th linear layer is... by Divide the activation or weights of the i-th linear layer into two parts, The central area is the region on both sides. and As a surrounding segment; the activations or weights in the target model are quantized to low precision, and during the quantization process, the quantization interval of the activations or weights located in the central segment is smaller than the quantization interval of the activations or weights in the surrounding segment.

[0027] The loss calculation module is used to simultaneously calculate the full-precision forward propagation result of activations or weights and the quantized low-precision forward propagation result during the forward propagation of the target model in the t-th training iteration. Based on the full-precision forward propagation result and the low-precision forward propagation result, the quantization error of each layer of the target model is obtained. The quantization error of each layer of the target model is summed, multiplied by a preset coefficient, and added to the loss function of the target model training result to obtain the final loss function.

[0028] a task execution module, configured to determine whether the final loss function is less than a preset value or t reaches a preset iteration number, if yes, save the current target model as an execution model, and save the current segmentation division parameter use the current segmentation division parameter quantize the activation or weight of the execution model to obtain a final model, and execute a corresponding task by using the final model; otherwise, increase t by 1, and call the iteration training module again.

[0029] The deep neural network adaptive quantization system based on joint optimization, wherein the target model is an image classification deep neural network model, a target picture to be classified is input into the final model, and the final model executes an image classification task to obtain a category to which the target picture belongs.

[0030] The deep neural network adaptive quantization system based on joint optimization, wherein

[0031] The quantization interval of the activation or weight located in the center segment is:

[0032]

[0033] The quantization interval of the activation or weight located in the surrounding segment is:

[0034]

[0035] The deep neural network adaptive quantization system based on joint optimization, wherein the activation or weight located in the center segment is quantized as:

[0036]

[0037] The activation or weight located in the surrounding segment is quantized as:

[0038]

[0039] wherein is a full-precision activation or weight value of an i-th linear layer in the t-th iteration, is a low-precision fixed-point number of the i-th linear layer after quantization;

[0040] the quantization error of each linear layer of the target model is calculated as is summed, multiplied by a preset coefficient γ, and added to a loss function of a training result of the target model to obtain a final loss function; the gradient of the deep neural network parameter is calculated in the back propagation process, and the gradient of the segmentation division parameter is calculated The gradient of the segmentation division parameter calculated after the back propagation is ​​Then, after the tth iteration, the segment division parameters are updated, and the update formula is:

[0041]

[0042] where is the learning rate, and then use to calculate the (t+1)th iteration.

[0043] The application further provides a storage medium for storing a program for executing any one of the deep neural network adaptive quantization methods.

[0044] The application further provides a client for any one of the deep neural network adaptive quantization systems.

[0045] From the above scheme, the application has the following advantages:

[0046] Compared with the prior art, the application can better fit the data distribution of the deep neural network activation and weight, quantize the activation values and weight of all linear layers in the deep neural network into low bits, thereby reducing the quantization error and achieving a reasoning accuracy loss of less than 1%. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The application is a segment division schematic diagram;

[0048] Figure 2 The application is a method flowchart. DETAILED DESCRIPTION

[0049] When the inventors were researching deep neural network quantization acceleration, they found that the existing work uses uniform quantization or fixed format (such as floating-point number FP8) non-uniform quantization, and these quantization methods lack adaptability to different data.

[0050] The present application finds that the distribution of the activation (and weight) of the deep neural network is approximately Gaussian, which can be divided into a "central segment" and a "peripheral segment", and the characteristics of different parts are different. The data value of the "central segment" is small, the data amount is large, and the data distribution changes greatly; while the data value of the "peripheral segment" is large, the data amount is small, and the data distribution changes little. At the same time, the variance of the activation (and weight) distribution of the deep neural network of different layers is different, that is, the height and width of the "central segment" distribution and the length of the "peripheral segment" are different. If the fixed interval uniform quantization or the non-uniform quantization with fixed format (such as floating-point number FP8) is used as in the prior art, it will cause a large quantization error, which will damage the effect of the deep neural network. Therefore, when quantizing the deep neural network, in order to reduce the quantization error, it is necessary to adaptively adjust the quantization interval of different regions. However, how to allocate the quantization interval of different regions is a difficulty. If the number of divided regions is too large, the number of parameters to be optimized is too large, and it is difficult to find an optimal set of parameters.

[0051] The inventors have found that the optimal quantization interval allocation parameter can be adaptively realized by joint optimization. According to the shape characteristics of the activation and weight distribution of the deep neural network, the data distribution can be first divided into a "central segment" and a "peripheral segment". Then, in the quantization-aware training process, a suitable data distribution division parameter, i.e. a segmenting parameter, is adaptively learned, so that different division methods can be realized for different deep neural network distributions, thereby adaptively obtaining the most suitable quantization interval for the deep neural network.

[0052] In learning the division parameter, in order to organically combine the learning data distribution division parameter with the training deep neural network parameter and reduce the calculation overhead of training the deep neural network due to learning the division parameter, the inventors propose an adaptive quantization method based on joint optimization, which jointly optimizes the division parameter and the neural network parameter, so that the data division of the deep neural network and the training of the neural network parameter are performed together. Based on the above findings and thinking, the inventors propose an adaptive quantization method based on quantization training. Specifically, in order to realize the above technical effects, the present application includes the following key technical points:

[0053] Key point 1: The activation values (or weights) of each layer are adaptively divided into a "central segment" and a "peripheral segment", and different quantization intervals are used for quantization. The absolute value of the data of the "central segment" is small, the data amount is large, and a small quantization interval is used for uniform quantization; the absolute value of the data of the "peripheral segment" is large, the data amount is small, and a large quantization interval is used for quantization. The use of a small quantization interval for the "central segment" will reduce the quantization error; while the use of a large quantization interval for the "peripheral segment" will not increase the quantization error too much. Therefore, the overall quantization error can be reduced, and the problem of network precision degradation caused by the large quantization error of the existing quantization method can be partially solved.

[0054] Key point 2, adaptively obtain the segmentation division parameter by joint optimization. The optimal segmentation division parameter is selected by using the quantization error, and the optimization process is to add the quantization error multiplied by a coefficient to the network loss function for optimization by the network optimizer, wherein the specific form of the network loss function is not limited by the application, and any loss function related to measuring the deviation of the model training result from the target value can be used as the network loss function. By joint optimization, the segmentation division parameter is adaptively obtained, so that the quantization method better fits the different data distribution shapes of different deep neural networks, reduces the quantization error of the activation and weight, and partially solves the problem of network precision decline caused by excessive quantization error of the existing quantization method.

[0055] The quantization training method can be applied to train the commonly used image classification deep neural network model, such as the ResNet-50 model. The input model is image data, and the model will output the classification result of the corresponding image. After using the quantization training method, a quantization model with less precision loss can be obtained through joint adaptive quantization training, reducing the calculation amount of training, speeding up the model training speed, and reducing the energy consumption of training the model. After training is completed, the calculation amount of model use can also be reduced in the use stage of the model, the prediction speed of image classification is accelerated, the energy consumption of classification is reduced, and the space size used for model storage is reduced.

[0056] In order to make the above features and effects of the application more clear and easy to understand, the following embodiments are described in detail below, and the drawings are described as follows.

[0057] The adaptive quantization method based on joint optimization first defines a segmentation division parameter, defines the region in the activation and weight of the deep neural network as the "center segment", and uses a smaller quantization interval for uniform quantization; the part outside the "center segment" is defined as the "peripheral segment", and a larger quantization interval is used for uniform quantization. The specific division method is as shown in Figure 1 In this case, the "center segment" has small data values, large data volume, and large data distribution changes. After using a smaller quantization interval, the quantization error of the "center segment" will be smaller. The data values of the "peripheral segment" are large, the data volume is small, and the data distribution changes are small. Using a larger quantization interval will not increase too much quantization error. In summary, the adaptive quantization method based on joint optimization allocates the quantization interval originally used in the "peripheral segment" to the "center segment". Under the premise that the data distribution of the "peripheral segment" changes little and the data distribution of the "center segment" changes greatly, the reduced quantization error of the "center segment" is greater than the increased quantization error of the "peripheral segment". Therefore, the overall quantization error is reduced, thereby reducing the accuracy loss of the quantized deep neural network.

[0058] The segmentation division parameters are obtained by joint optimization of network parameters in the deep neural network training process. The optimization target of the segmentation division parameters is to minimize the quantization error MSE to ensure that the quantization error is small enough to reduce the resulting model precision result decline. For a deep neural network, the respective segmentation division parameters are learned for the activations and weights of each layer, i.e., each layer is provided with independent segmentation division parameters; the quantization error of each layer of the deep neural network is summed, multiplied by an artificially set coefficient, and added to the loss function of the deep neural network itself to serve as the final loss function for training of the deep neural network. In this way, in the training process, the segmentation division parameters are jointly optimized with the network parameters to finally obtain the "center segment" and "peripheral segment" division parameters that minimize the quantization error and best adapt to the activation and weight distribution shape. In the inference stage, the "center segment" and "peripheral segment" division of the activations and weights of the deep neural network are performed using the segmentation division parameters obtained in the training, and the respective quantization intervals are used for quantization. Quantization can compress the neural network model, and the benefits of quantization mainly include reduced computational complexity, resulting in faster calculation, power saving in training models, reduced energy consumption, and smaller model storage space.

[0059] The entire process of the adaptive quantization method based on joint optimization is shown in Figure 2 The adaptive quantization method based on joint optimization is described below by taking activations (activation values) as an example. The calculation method of weights is the same.

[0060] Before the deep neural network training starts, the maximum absolute value Z of the activations (or weights) of all layers, such as convolutional layers and linear layers of fully connected layers, is counted. The initial segmentation division parameters of the activations (or weights) of each layer are set. The initial value of the segmentation division parameters of the activations (or weights) of the i-th layer is set as For example, it can be set as 1 / 4 of the maximum absolute value Z.

[0061] In the t-th iteration process, the value of the segmentation division parameters of the activations (or weights) of the i-th layer at this time is The application divides the activations (or weights) of the i-th layer into two parts, and The region is defined as the "center segment", and the regions on both sides and are defined as the "peripheral segment". The "center segment" uses a smaller quantization interval

[0062]

[0063] The "peripheral segment" uses a larger quantization interval

[0064]

[0065] In the forward propagation process of the tth iteration, the full-precision forward propagation result and the quantized low-precision forward propagation result are calculated simultaneously.

[0066] The activation (or weight) value of the "central segment" is quantized as

[0067]

[0068] The activation (or weight) value of the "peripheral segment" is quantized as

[0069]

[0070] wherein is the full-precision activation (or weight) value of the ith layer in the tth iteration, is the low-precision fixed point number of the ith layer after quantization. Then, the low-precision fixed point number is used to calculate the convolution and full connection layer in the deep neural network.

[0071] After the forward propagation is completed, the quantization error of each layer of the deep neural network is calculated, and the quantization error of the ith layer is After being multiplied by a manually set coefficient γ, the quantization error is added to the loss function of the deep neural network itself to serve as the final loss function. Then, the back propagation process is calculated, and the gradient of the deep neural network parameter is calculated, and the gradient of the segmented division parameter is calculated. Suppose that the gradient of the segmented division parameter calculated after the back propagation is then the segmented division parameter is updated after the tth iteration, and the update formula is

[0072]

[0073] wherein η is the learning rate. Then, the t+1th iteration is calculated using After multiple iterations, the deep neural network converges, and the segmented division parameter value that can guarantee the network precision is finally obtained. In the inference of the deep neural network, the segmented division parameter is used for quantization.

[0074] The following is a system embodiment corresponding to the above method embodiment. The technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they will not be repeated here. Correspondingly, the technical details mentioned in this embodiment can also be applied in the above embodiment.

[0075] The application further provides a deep neural network adaptive quantization system based on joint optimization, which comprises:

[0076]

[0077] ​An initial module is configured to obtain a deep neural network model to be trained as a target model, and count a maximum absolute value Z of activations or weights in all linear layers of the target model; and initialize a segmented division parameter of activation or weight setting of each linear layer according to the maximum absolute value Z

[0078] An iterative training module is configured to, when the target model is trained for the t-th time, set the segmented division parameter value of the i-th linear layer of the target model to The activation or weight of the i-th linear layer is divided into two parts, and the activation or weight of is set as a center segment, and the activation or weight of is set as a surrounding segment; the activation or weight of the target model is quantized to low precision, and a quantization interval of the activation or weight located in the center segment is smaller than a quantization interval of the activation or weight located in the surrounding segment in the process of quantization; A loss calculation module is configured to, in a forward propagation process of the target model in the t-th time of training, simultaneously calculate a full-precision forward propagation result and a low-precision forward propagation result of the activation or weight after quantization; obtain a quantization error of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, sum the quantization error of each layer of the target model, multiply the sum by a preset coefficient, and add the loss function of the training result of the target model to obtain a final loss function;

[0079] A task execution module is configured to judge whether the final loss function is smaller than a preset value or t reaches a preset iteration number, if yes, save the current target model as an execution model, and save the current segmented division parameter

[0080] quantize the activation or weight of the execution model to obtain a final model, and execute a corresponding task by using the final model; otherwise, t is increased by 1, and the iterative training module is called again. The deep neural network adaptive quantization system based on joint optimization, wherein the target model is an image classification deep neural network model, a target image to be classified is input into the final model, and the final model executes an image classification task to obtain a category to which the target image belongs.

[0081] The deep neural network adaptive quantization system based on joint optimization, wherein

[0082] The quantization interval of the activation or weight located in the center segment is

[0083] The quantization interval of the activation or weight located in the center segment is

[0084]

[0085] The quantization interval of the activation or weight located in the center segment is​​

[0086]

[0087] the joint optimization-based deep neural network adaptive quantization system, wherein the activations or weights located in the central segment are quantized as:

[0088]

[0089] the activations or weights located in the peripheral segment are quantized as:

[0090]

[0091] wherein is the full-precision activation or weight value of the i-th linear layer in the t-th iteration, is the low-precision fixed-point number of the i-th linear layer after quantization;

[0092] the quantization error of each linear layer of the target model is calculated as summed up, multiplied by a preset coefficient γ, and added to the loss function of the training result of the target model to obtain a final loss function; the gradient of the deep neural network parameters is calculated in the back propagation process, and the gradient of the segmented division parameter is also calculated; the gradient of the segmented division parameter calculated after back propagation is then the segmented division parameter is updated after the t-th iteration, and the update formula is:

[0093]

[0094] wherein η is a learning rate, and then is used to calculate the (t+1)-th iteration.

[0095] The application further provides a storage medium for storing a program for executing any one of the deep neural network adaptive quantization methods.

[0096] The application further provides a client for any one of the deep neural network adaptive quantization systems.

Claims

1. A deep neural network adaptive quantization method based on joint optimization, characterized in that, include: Step 1: Obtain the deep neural network model to be trained as the target model, and calculate the maximum absolute value of activations or weights in all linear layers of the target model. Based on the maximum absolute value Initialize the segmentation parameters for the activation or weight settings of each linear layer. ; Step 2, when the target model... During the training iteration, the first... The piecewise partitioning parameter value for the activation or weights of the linear layer is... ;by The first The activation or weights of the linear layer are divided into two parts, The central area is the region on both sides. and As a surrounding segment; the activations or weights in the target model are quantized to low precision, and during the quantization process, the quantization interval of the activations or weights located in the central segment is smaller than the quantization interval of the activations or weights in the surrounding segment. Step 3, in the target model During the forward propagation process of each training iteration, the full-precision forward propagation result of the activations or weights and the quantized low-precision forward propagation result are calculated simultaneously. Based on the full-precision forward propagation result and the low-precision forward propagation result, the quantization error of each layer of the target model is obtained. The quantization error of each layer of the target model is summed, multiplied by a preset coefficient, and added to the loss function of the target model training result to obtain the final loss function. Step 4: Determine whether the final loss function is less than the preset value or If the preset number of iterations is reached, the current target model is saved as the execution model, and the current segmentation parameters are also saved. Use the current segmentation parameters The activation or weights of the execution model are quantified to obtain the final model, which is then used to execute the corresponding task. Otherwise, increment t by 1 and execute step 2 again; The target model is a deep neural network model for image classification. The target image to be classified is input into the final model, and the final model performs the image classification task to obtain the category to which the target image belongs. The quantization interval for activations or weights located in this central segment is: The quantization interval for activations or weights located in this surrounding segment is: 。 2. The deep neural network adaptive quantization method based on joint optimization as described in claim 1, characterized in that, The activations or weights located in this central segment are quantized as follows: The activations or weights located in this surrounding segment are quantized as follows: in It is the first In the nth iteration Full-precision activation or weight values ​​for linear layers. It is the quantified first Low-precision fixed-point number of linear layers; The quantization error of each linear layer of the target model Summation, multiplied by a preset coefficient The loss function after training the target model The summation yields the final loss function; the backpropagation process calculates the gradients of the deep neural network parameters and simultaneously calculates the segmentation parameters. The gradient; the segmentation parameters are calculated after backpropagation. The gradient is Then the first After the next iteration, the segmentation parameters are... The update is performed using the following formula: in It's the learning rate, which will be used later. Calculate the first The next iteration.

3. A deep neural network adaptive quantization system based on joint optimization, characterized in that, include: The initial module is used to obtain the deep neural network model to be trained as the target model, and to calculate the maximum absolute value of the activations or weights in all linear layers of the target model. Based on the maximum absolute value Initialize the segmentation parameters for the activation or weight settings of each linear layer. ; Iterative training module, when the target model... During the training iteration, the first... The piecewise partitioning parameter value for the activation or weights of the linear layer is... ;by The first The activation or weights of the linear layer are divided into two parts, The central area is the region on both sides. and As a surrounding segment; the activations or weights in the target model are quantized to low precision, and during the quantization process, the quantization interval of the activations or weights located in the central segment is smaller than the quantization interval of the activations or weights in the surrounding segment. The loss calculation module is used in the target model at the... During the forward propagation process of each training iteration, the full-precision forward propagation result of the activations or weights and the quantized low-precision forward propagation result are calculated simultaneously. Based on the full-precision forward propagation result and the low-precision forward propagation result, the quantization error of each layer of the target model is obtained. The quantization error of each layer of the target model is summed, multiplied by a preset coefficient, and added to the loss function of the target model training result to obtain the final loss function. The task execution module is used to determine whether the final loss function is less than a preset value or If the preset number of iterations is reached, the current target model is saved as the execution model, and the current segmentation parameters are also saved. Use the current segmentation parameters The activation or weights of the execution model are quantified to obtain the final model, which is then used to execute the corresponding task. Otherwise, increment t by 1 and call the iterative training module again; The target model is a deep neural network model for image classification. The target image to be classified is input into the final model, and the final model performs the image classification task to obtain the category to which the target image belongs. The quantization interval for activations or weights located in this central segment is: The quantization interval for activations or weights located in this surrounding segment is: 。 4. The deep neural network adaptive quantization system based on joint optimization as described in claim 3, characterized in that, The activations or weights located in this central segment are quantized as follows: The activations or weights located in this surrounding segment are quantized as follows: in It is the first In the nth iteration Full-precision activation or weight values ​​for linear layers. It is the quantified first Low-precision fixed-point number of linear layers; The quantization error of each linear layer of the target model Summation, multiplied by a preset coefficient The loss function after training the target model The summation yields the final loss function; the backpropagation process calculates the gradients of the deep neural network parameters and simultaneously calculates the segmentation parameters. The gradient; the segmentation parameters are calculated after backpropagation. The gradient is Then the first After the next iteration, the segmentation parameters are... The update is performed using the following formula: in It's the learning rate, which will be used later. Calculate the first The next iteration.

5. A storage medium for storing a program that performs the deep neural network adaptive quantization method as described in claim 1 or 2.

6. A client for the deep neural network adaptive quantization system of claim 3 or 4.

Images