Neural network model quantification method and electronic device
By efficiently evaluating and selecting hybrid precision quantized configurations, the specific steps include quantization and layer sensitivity calculation, solving the problem of time-consuming selection of appropriate configurations and improving the efficiency of neural network models.
Patent Information
- Application Number
- CN202411457391.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2025-02-11
AI Technical Summary
Selecting the appropriate hybrid precision quantization configuration to balance the neural network model takes a long time to process accuracy and speed in multimedia data.
By efficiently selecting and evaluating the configuration of hybrid precision quantization, specific steps include performing quantization on multiple layers of the neural network model, calculating layer sensitivity, and canceling quantization of layers with the highest layer sensitivity layer by layer until a predetermined number of To achieve the neural network model, the best neural network model is selected.
Improves the efficiency of selecting the appropriate configuration from all hybrid precision quantization configurations, reducing the time required to determine the neural network model used to process multimedia data.
Smart Images
Figure CN119416838A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a quantization method and electronic device for a neural network model. Background Art
[0002] Neural network models are widely used to process multimedia data. Quantization (e.g., Mixed Precision Quantization (MPQ)) is one of the effective ways to improve the performance of neural network models and compress the size of neural network models. However, it may be very time-consuming to select a reasonable configuration from all mixed precision quantization configurations that can balance the accuracy and speed of neural network models in processing multimedia data. Summary of the invention
[0003] The present disclosure provides a quantization method and electronic device for a neural network model. In some embodiments, the method efficiently determines a neural network model for processing multimedia data, for example, by efficiently selecting and evaluating a configuration of mixed precision quantization.
[0004] In a first general aspect, a quantization method for a neural network model for processing multimedia data includes: (i) performing quantization on multiple layers in a first neural network model to obtain a second neural network model; (ii) using the second neural network model as a current neural network model to calculate the layer sensitivity of each of the multiple layers; (iii) updating the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers in the current neural network model that have been quantized to obtain a third neural network model; (iv) using the third neural network model as the current neural network model to calculate the layer sensitivity of each layer in the layers in the current neural network model that have been quantized; (v) repeating steps (iii) and (iv) until a predetermined number of third neural network models are obtained; and (vi) selecting the best neural network model from among the predetermined number of third neural network models.
[0005] In some embodiments, a mixed precision quantization configuration is found by greedily searching for the layer with the highest layer sensitivity among the layers on which quantization is performed and restoring the layer with the highest layer sensitivity, thereby improving the efficiency of selecting a suitable configuration from all mixed precision quantization configurations to efficiently determine a neural network model for processing multimedia data.
[0006] The step of canceling quantization of the layer with the highest layer sensitivity includes setting the weight of the layer with the highest layer sensitivity to the weight that the layer with the highest layer sensitivity had before quantization, wherein the predetermined number is the number of the multiple layers.
[0007] The step of calculating the layer sensitivity of each layer of the multiple layers using the second neural network model as the current neural network model includes: determining the first layer sensitivity of the corresponding layer based on the position of the corresponding layer in the current neural network model; determining the second layer sensitivity of the corresponding layer based on the noise change rate of the corresponding layer; determining the layer sensitivity of the corresponding layer based on the first layer sensitivity and the second layer sensitivity.
[0008] The step of determining the second layer sensitivity of the corresponding layer includes: determining the input noise-to-signal ratio of the corresponding layer based on the original input and quantized input of the corresponding layer; determining the output noise-to-signal ratio of the corresponding layer based on the original output and quantized output of the corresponding layer; determining the noise change rate of the corresponding layer based on the input noise-to-signal ratio and the output noise-to-signal ratio of the corresponding layer, wherein the original input of the corresponding layer represents the input of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized input of the corresponding layer represents the input of the corresponding layer obtained by executing the current neural network model, wherein the original output of the corresponding layer represents the output of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized output of the corresponding layer represents the output of the corresponding layer obtained by executing the current neural network model.
[0009] The distance of the corresponding layer from the input layer of the current neural network model is inversely proportional to the first-layer sensitivity of the corresponding layer, wherein the first-layer sensitivity of the corresponding layer is determined based on the range of inverse quantization weights of each layer after the corresponding layer in the current neural network model, wherein the inverse quantization weights of each layer after the corresponding layer in the current neural network model are based on the weights of each layer after the corresponding layer in the current neural network model after quantization is performed.
[0010] In some embodiments, by determining the layer sensitivity using a first layer sensitivity that reflects the influence of the layer position and a second layer sensitivity that reflects the influence of quantization noise on each layer, the sensitivity of the layer to quantization can be more effectively measured, and the decrease in the accuracy of the neural network model caused by quantizing a layer with high sensitivity can be avoided. Therefore, a suitable mixed precision quantization configuration can be selected.
[0011] The step of selecting the best neural network model may include: calculating the loss associated with each of a predetermined number of third neural network models based on the noise change rate of the output layer of each of the predetermined number of third neural network models and the size of each of the predetermined number of third neural network models; and selecting a neural network model with the smallest loss from among the predetermined number of third neural network models.
[0012] The step of selecting the optimal neural network model may include determining a size of each of the predetermined number of third neural network models based on a sum of bit widths of weights of a plurality of layers of each of the predetermined number of third neural network models.
[0013] In some embodiments, using the noise change rate of the output layer of the third neural network model and the size of the third neural network model as indicators for evaluating the configuration of the third neural network model or mixed precision quantization can reduce the amount of multimedia data used for evaluation while maintaining the accuracy of the evaluation, and reduce the time and power consumption required for the evaluation.
[0014] In a second general aspect, an electronic device includes: a quantization module configured to: perform quantization on multiple layers in a first neural network model to obtain a second neural network model; an update module configured to: (i) use the second neural network model as a current neural network model to calculate the layer sensitivity of each of the multiple layers, (ii) update the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers in the current neural network model that have been quantized to obtain a third neural network model, (iii) use the third neural network model as the current neural network model to calculate the layer sensitivity of each layer in the layers in the current neural network model that have been quantized, and (iv) repeat steps (ii) and (iii) until a predetermined number of third neural network models are obtained; and a selection module configured to: select the best neural network model from among the predetermined number of third neural network models.
[0015] In some embodiments, a mixed precision quantization configuration is found by greedily searching for the layer with the highest layer sensitivity among the layers on which quantization is performed and restoring the layer with the highest layer sensitivity, thereby improving the efficiency of selecting a suitable configuration from all mixed precision quantization configurations to efficiently determine a neural network model for processing multimedia data.
[0016] The update module is configured to un-quantize the layer having the highest layer sensitivity by setting the weight of the layer having the highest layer sensitivity to the weight before quantization, wherein the predetermined number is equal to the number of the plurality of layers.
[0017] The update module is configured to calculate the layer sensitivity of the corresponding layer among the multiple layers through the following steps: determining the first layer sensitivity of the corresponding layer based on the position of the corresponding layer in the current neural network model; determining the second layer sensitivity of the corresponding layer based on the noise change rate of the corresponding layer; determining the layer sensitivity of the corresponding layer based on the first layer sensitivity and the second layer sensitivity.
[0018] The update module is configured to: determine the input noise-to-signal ratio of the corresponding layer based on the original input and quantized input of the corresponding layer; determine the output noise-to-signal ratio of the corresponding layer based on the original output and quantized output of the corresponding layer; determine the noise change rate of the corresponding layer based on the input noise-to-signal ratio and the output noise-to-signal ratio of the corresponding layer, wherein the original input of the corresponding layer represents the input of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized input of the corresponding layer represents the input of the corresponding layer obtained by executing the current neural network model, wherein the original output of the corresponding layer represents the output of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized output of the corresponding layer represents the output of the corresponding layer obtained by executing the current neural network model.
[0019] The distance of the corresponding layer from the input layer of the current neural network model is inversely proportional to the first-layer sensitivity of the corresponding layer, wherein the first-layer sensitivity of the corresponding layer is determined based on the range of inverse quantization weights of each layer after the corresponding layer in the current neural network model, wherein the inverse quantization weights of each layer after the corresponding layer in the current neural network model are based on the weights of each layer after the corresponding layer in the current neural network model after quantization is performed.
[0020] In some embodiments, by determining the layer sensitivity using a first layer sensitivity that reflects the influence of the layer position and a second layer sensitivity that reflects the influence of quantization noise on each layer, the sensitivity of the layer to quantization can be more effectively measured, and the decrease in the accuracy of the neural network model caused by quantizing a layer with high sensitivity can be avoided. Therefore, a suitable mixed precision quantization configuration can be selected.
[0021] The selection module can be configured to: calculate the loss associated with each of the predetermined number of third neural network models based on the noise change rate of the output layer of each of the predetermined number of third neural network models and the size of each of the predetermined number of third neural network models; and select the neural network model with the smallest loss from among the predetermined number of third neural network models.
[0022] The selection module may be configured to determine a size of each of the predetermined number of third neural network models based on a sum of bit widths of weights of a plurality of layers of each of the predetermined number of third neural network models.
[0023] In some embodiments, using the noise change rate of the output layer of the third neural network model and the size of the third neural network model as indicators for evaluating the configuration of the third neural network model or mixed precision quantization can reduce the amount of multimedia data used for evaluation while maintaining the accuracy of the evaluation, and reduce the time and power consumption required for the evaluation.
[0024] In a third general aspect, a computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor performs the following operations: (i) quantize multiple layers in a first neural network model to obtain a second neural network model; (ii) use the second neural network model as a current neural network model to calculate the layer sensitivity of each of the multiple layers; (iii) update the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers in the current neural network model that have been quantized to obtain a third neural network model; (iv) use the third neural network model as the current neural network model to calculate the layer sensitivity of each layer in the layers in the current neural network model that have been quantized; (v) repeat steps (iii) and (iv) until a predetermined number of third neural network models are obtained; and (vi) select the best neural network model from the predetermined number of third neural network models.
[0025] The step of canceling quantization of the layer with the highest layer sensitivity includes setting the weight of the layer with the highest layer sensitivity to the weight that the layer with the highest layer sensitivity had before quantization, wherein the predetermined number is the number of the multiple layers.
[0026] The step of calculating the layer sensitivity of each layer of the multiple layers using the second neural network model as the current neural network model includes: determining the first layer sensitivity of the corresponding layer based on the position of the corresponding layer in the current neural network model; determining the second layer sensitivity of the corresponding layer based on the noise change rate of the corresponding layer; determining the layer sensitivity of the corresponding layer based on the first layer sensitivity and the second layer sensitivity.
[0027] The step of determining the second layer sensitivity of the corresponding layer includes: determining the input noise-to-signal ratio of the corresponding layer based on the original input and quantized input of the corresponding layer; determining the output noise-to-signal ratio of the corresponding layer based on the original output and quantized output of the corresponding layer; determining the noise change rate of the corresponding layer based on the input noise-to-signal ratio and the output noise-to-signal ratio of the corresponding layer, wherein the original input of the corresponding layer represents the input of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized input of the corresponding layer represents the input of the corresponding layer obtained by executing the current neural network model, wherein the original output of the corresponding layer represents the output of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized output of the corresponding layer represents the output of the corresponding layer obtained by executing the current neural network model.
[0028] The step of selecting the best neural network model includes: calculating the loss associated with each of a predetermined number of third neural network models based on the noise change rate of the output layer of each of the predetermined number of third neural network models and the size of each of the predetermined number of third neural network models; and selecting the neural network model with the smallest loss from among the predetermined number of third neural network models.
[0029] The step of selecting the best neural network model includes determining the size of each of the predetermined number of third neural network models based on the sum of the bit widths of weights of multiple layers of each of the predetermined number of third neural network models. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a diagram illustrating an example of a neural network model.
[0031] Figure 2 is a flowchart illustrating an example of a quantization method for a neural network model.
[0032] Figure 3 is a flow chart illustrating an example of a process for determining layer sensitivity.
[0033] Figure 4 is a flowchart illustrating an example of a quantization method for a neural network model.
[0034] Figure 5 is a block diagram illustrating an example of an electronic device. DETAILED DESCRIPTION
[0035] Various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be apparent upon understanding the disclosure of the present application. For example, except for operations that must occur in a particular order, the order of operations described herein is merely exemplary and is not limited to those orders set forth herein, but may be changed as will be apparent upon understanding the disclosure of the present application. In addition, descriptions of features known in the art may be omitted for greater clarity and brevity.
[0036] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.
[0037] The structural or functional description of the examples disclosed here below is intended only for the purpose of describing the examples, and the examples may be implemented in various forms. The examples are not intended to be limiting, but also support various modifications, equivalents and substitutes within the scope of the claims.
[0038] Although the terms "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another component. For example, within the scope of the rights according to the concept of the present disclosure, a "first" component may be referred to as a "second" component, or similarly, a "second" component may be referred to as a "first" component.
[0039] It will be understood that when a component is referred to as being “connected to” another component, the component may be directly connected or coupled to the other component or intervening components may be present.
[0040] As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It should also be understood that when the terms "include" and / or "comprising" are used in this specification, it indicates the presence of stated features, integers, steps, operations, elements, components or combinations thereof, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0041] Unless otherwise defined, all terms (including technical or scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the examples belong. It will also be understood that, unless expressly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense.
[0042] Hereinafter, examples will be described in detail with reference to the accompanying drawings. Regarding reference numerals assigned to elements in the drawings, it should be noted that the same elements will be denoted by the same reference numerals and redundant descriptions thereof will be omitted.
[0043] Figure 1 is a diagram illustrating an example of a neural network model.
[0044] For example, Figure 1 is a diagram schematically showing the structure of a deep neural network (DNN) 10 as an example of a neural network model.
[0045] A neural network model may represent a computing system that focuses on biological neural networks that constitute an animal brain. Unlike classical algorithms that perform tasks according to predefined conditions (such as rule-based programming), a neural network model can be trained to perform tasks by considering multiple samples (or examples). A neural network model may have a structure in which artificial neurons (or neurons) are connected, and the connections between neurons may be referred to as synapses. A neuron may process a received signal and send the processed signal to another neuron through a synapse. The output of a neuron may be referred to as "activation". Neurons and / or synapses may have variable weights, and the influence of a signal processed by a neuron may be increased or decreased according to the weight. In particular, the weight associated with a single neuron may be referred to as a bias.
[0046] A deep neural network (DNN) or deep learning architecture may have a layer structure, and the output of a specific layer may be the input of a subsequent layer. In such a multi-layer structure, each layer may be trained based on multiple samples. A neural network model (such as a DNN) may be implemented by multiple nodes corresponding to neurons, respectively, which may require higher computational complexity to obtain good results (such as higher accuracy results), and therefore many computing resources may be required.
[0047] Reference Figure 1 , the DNN 10 may include a plurality of layers L1, L2, L3 ... LN (for example, N may be an integer greater than 3), and the output of the layer may be input to the subsequent layer through at least one channel. For example, the first layer L1 may provide the output to the second layer L2 through a plurality of channels CH11 ... CH1x by processing the sample SAM, and the second layer L2 may also provide the output to the third layer L3 through a plurality of channels CH21 ... CH2y. Finally, the Nth layer LN may output the result RES, and the result RES may include at least one value related to the sample SAM. The number of channels through which the outputs of the plurality of layers L1, L2, L3 ... LN are transmitted may be the same or different. For example, the number of channels CH21 ... CH2y of the second layer L2 and the number of channels CH31 ... CH3z of the third layer L3 may be the same or different. For example, x, y, and z may be integers greater than 1.
[0048] The sample SAM may be input data processed by the DNN 10. For example, the sample SAM may include multimedia data (such as but not limited to image data, voice data, text data, etc.).
[0049] The DNN 10 may include a large number of layers or channels, and therefore, the computational complexity of the DNN 10 may increase. A DNN 10 having a high computational complexity may require a large amount of resources. Therefore, in order to reduce the computational complexity of the DNN 10, the DNN 10 may be quantized. Quantization of the DNN 10 may refer to a process of mapping an input value to a number of values smaller than the number of input values (such as mapping a real number to an integer by rounding). The quantized DNN 10 may have a low computational complexity, but may have reduced accuracy due to errors occurring in the quantization process.
[0050] Figure 2 is a flowchart illustrating an example of a quantization method for a neural network model.
[0051] like Figure 2 As shown in FIG. 1 , in step S110 , quantization may be performed on multiple layers in the first neural network model to obtain a second neural network model. The second neural network model includes layers on which quantization has been performed.
[0052] In some embodiments, the first neural network model may be a high-precision model, and the second neural network model may be a low-precision model. For example, each layer in the first neural network model (e.g., the weight of each layer) may have a first bit width, each layer in the second neural network model (e.g., the weight of each layer) may have a second bit width, and the first bit width may be greater than the second bit width.
[0053] In step S120, the second neural network model can be used as the current neural network model to calculate the layer sensitivity of each layer in multiple layers (for example, the layers in the current neural network model on which quantization is performed). In this specification, the current neural network refers to the neural network model updated at the current moment or at the current step in the process of updating the first neural network model.
[0054] In some embodiments, the layer sensitivity of the layer on which quantization is performed may indicate the influence of quantization on the accuracy of the layer. For example, whether to cancel the quantization of the layer may be determined based on the layer sensitivity of the layer on which quantization is performed. For example, the higher the layer sensitivity of a layer (for example, the greater the influence of quantization on the accuracy of the layer), the higher the probability that the quantization of the layer is canceled (for example, the lower the probability that the layer is quantized in the optimal neural network model (to be described later)).
[0055] In one example, the layer sensitivity of the layer on which quantization is performed may be determined based on a first layer sensitivity (to be described later) and / or a second layer sensitivity of the layer on which quantization is performed.
[0056] In one example, the first layer sensitivity of the layer on which quantization is performed may be based on the position of the layer on which quantization is performed in the current neural network model.
[0057] In one example, the current neural network model and the first neural network model may be executed with predetermined input data, and based on the results of executing the current neural network model and the first neural network model, the second layer sensitivity of the layer in the current neural network model on which quantization is performed may be determined. For example, the second layer sensitivity of the layer on which quantization is performed may be determined based on the input and output of the layer on which quantization is performed in the current neural network model obtained after executing the current neural network, and the input and output of the layer corresponding to the layer on which quantization is performed in the first neural network model obtained after executing the first neural network model. For example, the predetermined input data may be multimedia data.
[0058] In the following, reference will be made to Figure 3 The process of determining layer sensitivity is described in detail.
[0059] In step S130, the current neural network model may be updated by canceling the quantization of the layer with the highest layer sensitivity among the layers to which quantization is performed in the current neural network model to obtain a third neural network model.
[0060] In some embodiments, a greedy search algorithm may be used to search for a layer with the highest layer sensitivity from among the layers on which quantization is performed, so as to improve search efficiency.
[0061] In some embodiments, the quantization of the layer with the highest layer sensitivity can be canceled by setting the weight of the layer with the highest layer sensitivity to the weight that the layer with the highest layer sensitivity had before quantization (e.g., before step S110) (e.g., the weight of the layer of the first neural network model corresponding to the layer with the highest layer sensitivity). In other words, the weight of the layer with the highest layer sensitivity before quantization is equal to the first set of values, and the weight of the layer with the highest layer sensitivity after quantization is equal to the second set of values. Quantization can be canceled by changing the second set of values back to the first set of values.
[0062] In step S140, the third neural network model may be used as the current neural network model to calculate the layer sensitivity of each layer in the layers on which quantization is performed in the current neural network model.
[0063] After step S150, the process may return to step S130 to repeat the step of updating the current neural network model (step S130) and the step of calculating the layer sensitivity (step S140) until a predetermined number of third neural network models are obtained.
[0064] In some embodiments, the predetermined number may be the number of multiple layers in the first neural network model. In other words, the above steps S130 and S140 may be repeated until the quantization of all layers in the second neural network model is canceled.
[0065] In step S160, an optimal neural network model may be selected from a predetermined number of third neural network models.
[0066] In some embodiments, the loss associated with the third neural network model may be determined based on the accuracy and size of the third neural network model. For example, the accuracy of the third neural network model may be represented as the noise change rate of the output layer of the third neural network model (which will be referred to below). Figure 3 Detailed description), the size of the third neural network model can be represented as the sum of the bit widths of the weights of multiple layers in the third neural network model.
[0067] In one example, the loss associated with the third neural network model can be determined by the following equation (1).
[0068] loss(c)=NSR(last_layer; c) + λ* model_size(c) Equation (1)
[0069] In equation (1), loss(c) may represent the loss associated with the third neural network model c, NSR(last_layer; c) may represent the noise change rate of the output layer in the third neural network model c, model_size(c) may represent the size of the third neural network model c, and λ is a predetermined value.
[0070] In one example, after determining the best neural network model, a mixed precision quantization configuration corresponding to the best neural network model (including the bit width of the weights of each layer in the best neural network model) may be determined.
[0071] In some embodiments, the mixed precision quantization configuration is found by greedily searching the layer with the highest layer sensitivity among the layers on which quantization is performed and restoring the layer with the highest layer sensitivity, thereby improving the efficiency of selecting a suitable configuration from all mixed precision quantization configurations.
[0072] Compared with the traditional way of evaluating a neural network model using a large amount of multimedia data (e.g., tens of thousands of pictures), the present disclosure uses the noise change rate of the output layer of the third neural network model and the size of the third neural network model as indicators for evaluating the configuration of the third neural network model or mixed precision quantization. As a result, the third neural network model can be evaluated with only a small amount of multimedia data (e.g., dozens of pictures), thereby effectively reducing the amount of multimedia data used for evaluation while maintaining the accuracy of the evaluation, and reducing the time and power consumption required for the evaluation.
[0073] Figure 3 is a flow chart illustrating an example of a process for determining layer sensitivity.
[0074] like Figure 3 As shown in, in step S210, the first layer sensitivity of the quantization layer can be determined based on the position of the layer that has been quantized (hereinafter referred to as the quantized layer or the quantized layer or the corresponding layer) in the current neural network model.
[0075] In some embodiments, the smaller the distance between the quantization layer in the current neural network model and the input layer of the current neural network model, the higher the first layer sensitivity of the quantization layer. For example, the distance between the quantization layer in the current neural network model and the input layer of the current neural network model is inversely proportional to the first layer sensitivity of the quantization layer.
[0076] In some embodiments, the first layer sensitivity of the quantization layer may be determined based on the range of the inverse quantization weights of each layer after the quantization layer in the current neural network model. For example, the first layer sensitivity of the quantization layer in the current neural network model may be the product of the range of the inverse quantization weights of each layer after the quantization layer in the current neural network model.
[0077] In some embodiments, the dequantized weight of each layer after the quantized layer in the current neural network model is based on the value of the weight of each layer after the quantized layer in the current neural network model after quantization is performed. That is, the dequantized weight of each layer after the quantized layer in the current neural network model is based on the weight of the layer in the second neural network model corresponding to each layer after the quantized layer in the current neural network model. For example, the dequantized weight of a specific layer in the current neural network model may be the weight of the layer in the second neural network model corresponding to the specific layer. In one example, the dequantized weight of the k-th layer in the current neural network model may be the weight of the k-th layer in the second neural network model.
[0078] In one example, the first layer sensitivity of the quantization layer may be calculated by the following equation (2).
[0079]
[0080] In equation (2), IQN(k) may represent the first layer sensitivity of the kth layer as the quantization layer in the current neural network model, N may represent the number of multiple layers in the current neural network model, and w dq,i It can represent the inverse quantization weight of the i-th layer in the current neural network model, range(w dq,i ) may represent the range of the inverse quantization weights of the i-th layer in the current neural network model. In one example, the distance between the quantization layer in the current neural network model and the input layer of the current neural network model may represent the number of layers between the quantization layer and the input layer.
[0081] However, the example is not limited to this, and other methods can be used to calculate the first layer sensitivity of the quantization layer in the current neural network model based on the range of the inverse quantization weights of each layer after the quantization layer in the current neural network model.
[0082] In one example, after obtaining the second neural network model by quantizing the first neural network model, the first layer sensitivity of each layer in the second neural network model may be calculated. For example, for each layer in the second neural network model, the product of the range of weights of each layer after the layer may be determined as the first layer sensitivity of the layer in the second neural network model. For example, the product of the range of weights of each layer after the kth layer in the second neural network model may be determined as the first layer sensitivity of the kth layer in the second neural network model.
[0083] In this case, the first layer sensitivity of the quantization layer in the current neural network model may be determined as the first layer sensitivity of the layer corresponding to the quantization layer in the second neural network model.
[0084] In step S220, the input-to-noise ratio of the quantization layer in the current neural network model may be determined based on the original input and the quantization input of the quantization layer in the current neural network model.
[0085] In one example, the original input of the quantization layer in the current neural network model may represent the input of the layer corresponding to the quantization layer in the first neural network model obtained by executing the first neural network model, and the quantization input of the quantization layer in the current neural network model represents the input of the quantization layer obtained by executing the current neural network model.
[0086] In this case, the input data used to execute the current neural network model may be the same as the input data used to execute the first neural network model, and may be predetermined multimedia data.
[0087] In one example, the input noise signal ratio of the quantization layer may be calculated by the following equation (3):
[0088]
[0089] In equation (3), NSR(kx) may represent the input noise signal ratio of the kth layer as the quantization layer in the current neural network model, M may represent the amount of data in the input of the kth layer in the current neural network model, and x noise,i It can represent the quantization noise of the i-th data in the input of the k-th layer in the current neural network model, x i It can represent the data corresponding to the i-th data in the input of the k-th layer in the first neural network model.
[0090] In one example, the input of the kth layer in the current neural network model may include (e.g., when the current neural network model is executed with predetermined input data) multiple data input to multiple nodes of the kth layer in the current neural network model. In one example, at least one data may be input to each node of the kth layer in the current neural network model.
[0091] In one example, the input of the kth layer in the first neural network model may include (e.g., when the first neural network model is executed with predetermined input data) multiple data input to multiple nodes of the kth layer in the first neural network model.
[0092] In one example, the quantization noise of the i-th data in the input of the k-th layer in the current neural network model may be the difference between the i-th data and the data corresponding to the i-th data in the input of the k-th layer in the first neural network model.
[0093] In step S230, the output signal-to-noise ratio of the quantization layer in the current neural network model may be determined based on the original output and the quantization output of the quantization layer in the current neural network model.
[0094] In one example, the original output of the quantization layer in the current neural network model may represent the output of the layer corresponding to the quantization layer in the first neural network model obtained by executing the first neural network model, and the quantization output of the quantization layer in the current neural network model represents the output of the quantization layer obtained by executing the current neural network model.
[0095] In one example, the output signal-to-noise ratio of the quantization layer may be calculated by the following equation (4).
[0096]
[0097] In equation (4), NSR(ky) may represent the output signal-to-noise ratio of the kth layer as the quantization layer in the current neural network model, L may represent the amount of data in the output of the kth layer in the current neural network model, and y noise,iIt can represent the quantization noise of the i-th data in the output of the k-th layer in the current neural network model, y i It can represent the data corresponding to the i-th data in the output of the k-th layer in the first neural network model.
[0098] In one example, the output of the kth layer in the current neural network model may include a plurality of data (e.g., activation values) output from a plurality of nodes of the kth layer in the current neural network model.
[0099] In one example, the output of the kth layer in the first neural network model may include a plurality of data (e.g., activation values) output from a plurality of nodes of the kth layer in the first neural network model.
[0100] In one example, the quantization noise of the i-th data in the output of the k-th layer in the current neural network model may be the difference between the i-th data and “the data corresponding to the i-th data in the output of the k-th layer in the first neural network model”.
[0101] For example, the quantization noise of the i-th data in the output of the k-th layer in the current neural network model (for example, the activation value output by the i-th node of the k-th layer) can be the difference between the i-th data output by the i-th node of the k-th layer in the current neural network model and the data output by the i-th node of the k-th layer in the first neural network model.
[0102] However, the example is not limited to this, and other calculations may be used to calculate the input-to-noise ratio and the output-to-noise ratio of the quantization layer based on the original input, original output, quantized input, and quantized output of the quantization layer in the current neural network model.
[0103] In step S240, the noise change rate of the quantization layer in the current neural network model can be determined as the second layer sensitivity based on the input noise-to-signal ratio and the output noise-to-signal ratio of the quantization layer in the current neural network model.
[0104] In one example, the noise change rate of the quantization layer may be calculated by the following equation (5).
[0105] OINR(k) = NSR(ofm) k -NSR(ifm) k Equation (5)
[0106] In equation (5), OINR(k) can represent the noise change rate of the kth layer or the second layer sensitivity of the quantization layer in the current neural network model, and NSR(ofm) k It can represent the output noise-to-signal ratio of the kth layer in the current neural network model, NSR(ifm) kThe input noise-to-signal ratio of the kth layer in the current neural network model may be represented as the output noise-to-signal ratio of the k-1th layer in the current neural network model.
[0107] However, the example is not limited thereto, and other calculations may be used to calculate the second layer sensitivity based on the input signal-to-noise ratio and the output signal-to-noise ratio.
[0108] In step S250 , the layer sensitivity of the quantization layer may be determined based on the first layer sensitivity and the second layer sensitivity of the quantization layer.
[0109] For example, the layer sensitivity of the quantization layer may be determined based on the sum of the first layer sensitivity and the second layer sensitivity of the quantization layer.
[0110] However, the example is not limited thereto, and other calculations may be used to calculate the layer sensitivity of the quantized layer based on the first layer sensitivity and the second layer sensitivity of the quantized layer. In one example, the layer sensitivity of the quantized layer may be calculated based only on the second layer sensitivity of the quantized layer, without considering the first layer sensitivity of the quantized layer.
[0111] In the present application, the layer sensitivity is determined by using the first layer sensitivity that reflects the influence of the layer position and the second layer sensitivity that reflects the influence of quantization noise on each layer. This can more effectively measure the sensitivity of the layer to quantization and avoid the decrease in accuracy of the neural network model caused by quantizing layers with high sensitivity, so as to select a suitable mixed precision quantization configuration.
[0112] Figure 4 is a flowchart illustrating an example of a quantization method for a neural network model.
[0113] like Figure 4 As shown in , in step S310, quantization may be performed on multiple layers in the first neural network model to obtain a second neural network model.
[0114] In step S320, the first layer sensitivity of each layer in the second neural network model may be calculated. Figure 3 The process of calculating the first layer sensitivity has been described in detail, and therefore, its detailed description will be omitted here to avoid redundancy.
[0115] In step S330 , the variable i may be set to an initial value of 1.
[0116] In step S340, the layer sensitivity of each quantization layer in the second neural network model may be calculated. For example, the initial second neural network model may be used as the current neural network model to calculate the layer sensitivity of each quantization layer in the current neural network model.
[0117] In step S350, the second neural network model may be updated by canceling quantization of a quantization layer having the highest layer sensitivity to obtain and output a third neural network model.
[0118] For example, suppose the first neural network model can be represented as Figure 1 In the DNN 10 shown in , the first neural network model may include multiple layers L1, L2, L3...LN, the second neural network model may include multiple layers L1', L2', L3'...LN' obtained by quantizing the multiple layers L1, L2, L3...LN, and the Kth layer LK' among the multiple layers L1', L2', L3'...LN' of the second neural network model has the highest layer sensitivity, then in step S350, the second neural network model can be updated by canceling the quantization of the Kth layer LK', and the updated second neural network model can be output as a third neural network model (including multiple layers L1', L2', L3'...LK-1', LK, LK+1'...LN').
[0119] In step S360, it may be determined whether the variable i is greater than or equal to N. For example, N may represent the number of multiple layers in the first neural network model.
[0120] If the variable i is less than N, then in step S370 , the variable i may be increased by 1, and steps S340 to S360 may be repeated.
[0121] For example, the updated second neural network model can be used as the current neural network model, the layer sensitivity of each quantized layer in the current neural network model can be calculated, and the second neural network model can be continuously updated to output the re-updated second neural network model as a new third neural network model. For example, the updated second neural network model including multiple layers L1', L2', L3'...LK-1', LK, LK+1'...LN' can be used to determine the layer with the highest layer sensitivity among the multiple layers L1', L2', L3'...LK-1', LK+1'...LN', and the second neural network model can be continuously updated by canceling the quantization of the layer with the highest layer sensitivity.
[0122] When the variable i is greater than or equal to N, in step S380, the best neural network model may be selected from the output N third neural network models.
[0123] Figure 5 is a block diagram illustrating an example of an electronic device.
[0124] For example, the electronic device may include a desktop computer, a laptop computer, a tablet computer, a server system, etc. However, the present disclosure is not limited thereto, and the electronic device may be any electronic device having a function of processing multimedia data.
[0125] like Figure 5 As shown in , the electronic device 1 may include a quantization module 11 , an update module 12 and a selection module 13 .
[0126] In some embodiments, the quantization module 11 may perform quantization on multiple layers in the first neural network model to obtain a second neural network model.
[0127] In some embodiments, the first neural network model may be a high-precision model, and the second neural network model may be a low-precision model. For example, each layer in the first neural network model (e.g., the weight of each layer) may have a first bit width, each layer in the second neural network model may have a second bit width, and the first bit width may be greater than the second bit width.
[0128] In some embodiments, the updating module 12 may use the second neural network model as the current neural network model, and calculate the layer sensitivity of each layer in the layers on which quantization is performed in the current neural network model.
[0129] The above has been referred to Figure 3 The process of determining layer sensitivity is described in detail, and repeated description will be omitted here to avoid redundancy.
[0130] In some embodiments, the updating module 12 may update the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers on which quantization is performed to obtain a third neural network model.
[0131] In some embodiments, a greedy search algorithm may be used to search for a layer having the highest layer sensitivity from among the layers on which quantization is performed, so as to improve search efficiency.
[0132] In some embodiments, the quantization of the layer with the highest layer sensitivity may be undoed by setting the weight of the layer with the highest layer sensitivity to the weight of the layer with the highest layer sensitivity before quantization is performed on the layer.
[0133] In some embodiments, the updating module 12 may use the third neural network model as the current neural network model, and calculate the layer sensitivity of each layer in the layers on which quantization is performed in the current neural network model.
[0134] In some implementations, the updating module 12 may repeat the steps of updating the current neural network model and calculating the layer sensitivity until a predetermined number of third neural network models are obtained.
[0135] In some embodiments, the predetermined number may be equal to the number of multiple layers in the first neural network model. In other words, the above steps of updating the current neural network model and calculating the layer sensitivity may be repeated until all layers in the second neural network model that have been quantized are restored.
[0136] In some embodiments, the selection module 13 may select the best neural network model from among a predetermined number of third neural network models.
[0137] The above has been referred to Figure 2 The process of selecting the best neural network model is described in detail, and repeated descriptions will be omitted here to avoid redundancy.
[0138] The devices, units, modules and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or processors or computers can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. Hardware components can have any one or more of different processing configurations, and examples of different processing configurations include: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instructions single data (MISD) multiprocessing, and multiple instructions multiple data (MIMD) multiprocessing.
[0139] The method for performing the operations described in this application is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as executing instructions or software as described above to perform the operations performed by the method described in this application. For example, a single operation, or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation, or two or more operations.
[0140] The instructions or software for controlling a processor or computer to implement a hardware component and perform the method as described above may be written as a computer program, a code segment, an instruction, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or a special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by a processor or computer. In another example, the instructions or software include high-level code executed by a processor or computer using an interpreter. An ordinary programmer in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, and the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing operations performed by the hardware components and methods as described above.
[0141] The instructions or software for controlling a processor or computer to implement the hardware components and perform the methods described above, and any associated data, data files, and data structures are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), flash memory, card memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.
[0142] While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.
Claims
1. A quantization method for a neural network model, wherein the neural network model is used to process multimedia data, the quantization method comprising: (i) performing quantization on a plurality of layers in the first neural network model to obtain a second neural network model; (ii) using the second neural network model as the current neural network model, calculating the layer sensitivity of each layer in the plurality of layers; (iii) updating the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers to which quantization is performed in the current neural network model to obtain a third neural network model; (iv) using the third neural network model as the current neural network model, calculating the layer sensitivity of each layer in the layers in the current neural network model on which quantization is performed; (v) repeating step (iii) and step (iv) until a predetermined number of third neural network models are obtained; (vi) Selecting the best neural network model from among a predetermined number of third neural network models.
2. The method according to claim 1, wherein: The step of unquantizing the layer having the highest layer sensitivity includes setting the weight of the layer having the highest layer sensitivity to the weight that the layer having the highest layer sensitivity had before quantization, The predetermined number is the number of the plurality of layers.
3. The method according to claim 1, wherein: The step of calculating the layer sensitivity of each layer of the plurality of layers using the second neural network model as the current neural network model comprises: Determine the first layer sensitivity of the corresponding layer based on the position of the corresponding layer in the current neural network model; Determining a second layer sensitivity of the corresponding layer based on a noise change rate of the corresponding layer; Based on the first layer sensitivity and the second layer sensitivity, a layer sensitivity of a corresponding layer is determined.
4. The method according to claim 3, wherein: The steps of determining the second layer sensitivity of the corresponding layer include: Determine an input-to-noise signal ratio of the corresponding layer based on the original input and the quantized input of the corresponding layer; Determine an output signal-to-noise ratio of the corresponding layer based on the original output and the quantized output of the corresponding layer; Based on the input noise-to-signal ratio and output noise-to-signal ratio of the corresponding layer, the noise change rate of the corresponding layer is determined. The original input of the corresponding layer represents the input of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized input of the corresponding layer represents the input of the corresponding layer obtained by executing the current neural network model. Among them, the original output of the corresponding layer represents the output of the corresponding layer of the first neural network model corresponding to the corresponding layer obtained by executing the first neural network model, and the quantized output of the corresponding layer represents the output of the corresponding layer obtained by executing the current neural network model.
5. The method according to claim 3, wherein: The distance between the corresponding layer and the input layer of the current neural network model is inversely proportional to the sensitivity of the first layer of the corresponding layer. Wherein, based on the range of the inverse quantization weights of each layer after the corresponding layer in the current neural network model, the first layer sensitivity of the corresponding layer is determined, Among them, the dequantized weights of each subsequent layer of the corresponding layer in the current neural network model are based on the weights of each subsequent layer of the corresponding layer in the current neural network model after quantization is performed.
6. The method according to claim 1, wherein: The steps to select the best neural network model include: calculating a loss associated with each of the predetermined number of third neural network models based on a noise change rate of an output layer of each of the predetermined number of third neural network models and a size of each of the predetermined number of third neural network models; A neural network model with a minimum loss is selected from among a predetermined number of third neural network models.
7. The method according to claim 6, wherein: The step of selecting the best neural network model includes determining the size of each of the predetermined number of third neural network models based on the sum of the bit widths of weights of multiple layers of each of the predetermined number of third neural network models.
8. An electronic device comprising: A quantization module is configured to: perform quantization on multiple layers in the first neural network model to obtain a second neural network model; Update module, configured as: (i) using the second neural network model as the current neural network model, calculating the layer sensitivity of each layer in the plurality of layers, (ii) updating the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers to which quantization is performed in the current neural network model to obtain a third neural network model, (iii) using the third neural network model as the current neural network model, calculating the layer sensitivity of each layer in the layers in the current neural network model on which quantization is performed, and (iv) repeating step (ii) and step (iii) until a predetermined number of third neural network models are obtained; and The selection module is configured to select the best neural network model from a predetermined number of third neural network models.
9. The electronic device according to claim 8, wherein: The update module is configured to unquantize the layer having the highest layer sensitivity by setting the weight of the layer having the highest layer sensitivity to the weight before quantization, The predetermined number is equal to the number of the plurality of layers.
10. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the processor is caused to perform the following operations: (i) performing quantization on a plurality of layers in the first neural network model to obtain a second neural network model; (ii) using the second neural network model as the current neural network model, calculating the layer sensitivity of each layer in the plurality of layers; (iii) updating the current neural network model by canceling the quantization of the layer with the highest layer sensitivity among the layers to which quantization is performed in the current neural network model to obtain a third neural network model; (iv) using the third neural network model as the current neural network model, calculating the layer sensitivity of each layer in the layers in the current neural network model on which quantization is performed; (v) repeating step (iii) and step (iv) until a predetermined number of third neural network models are obtained; (vi) Selecting the best neural network model from among a predetermined number of third neural network models.