A mixed precision quantization method of a deep neural network and a related device

Abstract

The application belongs to the technical field of mixed precision quantization, and discloses a mixed precision quantization method for a deep neural network and a related device; wherein the mixed precision quantization method comprises: using a mutation strategy in a differential evolution algorithm to perform diversity processing on a population to obtain a mutated population; using a binomial crossover operation to recombine sub-components of a mutated individual in the mutated population and sub-components of a target individual in a current population to obtain a plurality of trial individuals; performing mixed precision quantization on the deep neural network based on the trial individuals and the target individual, and selecting a better individual from the trial individuals and the target individual; and judging whether the differential evolution algorithm converges based on the selected better individual, obtaining a final deep neural network after mixed precision quantization if the differential evolution algorithm converges, or using the selected better individual as a population of the next generation if the differential evolution algorithm does not converge.The application can realize mixed precision quantization of a deep neural network, and has the advantages of low computational complexity and strong adaptability.

Description

Technical Field

[0001] This invention belongs to the field of mixed-precision quantization technology, and specifically relates to a mixed-precision quantization method and related apparatus for deep neural networks. Background Technology

[0002] In deep learning, Convolutional Neural Networks (CNNs) are commonly used to solve object classification tasks. Interpretably, CNNs are a variant of Deep Neural Networks (DNN) architecture, widely used in machine vision tasks such as image classification and object detection, achieving great success. Neural networks are computationally and memory-intensive, characterized by large storage space requirements and high power consumption, making them difficult to deploy on embedded systems with limited hardware resources. Given these limitations, deep neural networks need to be lightweighted to reduce memory usage and computational requirements, enabling deployment in embedded devices and truly bringing their powerful classification and recognition performance to life in research and practical applications. Therefore, lightweighting deep neural networks is crucial for advancing deep neural network technology from theory to practice and improving the conversion rate of research results.

[0003] Explained, quantization is a technique for optimizing deep learning models. By converting model parameters and / or intermediate computation results from 32-bit floating-point numbers (single precision) to lower-bit integer representations, it reduces the model's storage requirements and computational costs. Common quantization techniques include binary quantization, ternary quantization, quaternary quantization, and fixed-point quantization. These techniques can significantly reduce the model's storage space and computational resource requirements, making it possible to deploy deep learning models on resource-constrained devices. Further explained, mixed-precision quantization is a specific form of quantization technique. Compared to the common quantization techniques mentioned above, mixed-precision quantization assigns different quantization bit widths to the parameters of each layer of the neural network, taking advantage of the inconsistent quantization sensitivity of the parameters across layers.

[0004] Existing mixed-precision quantization techniques suffer from high computational complexity and poor adaptability, necessitating a new neural network mixed-precision quantization technique to address these issues. Specifically, existing schemes using reinforcement learning for mixed-precision quantization of deep neural networks require extensive training and interaction; while existing schemes relying on model feature information are dependent on the specific structure and type of the neural network, resulting in poor adaptability. Summary of the Invention

[0005] The purpose of this invention is to provide a mixed-precision quantization method and related apparatus for deep neural networks to solve one or more of the aforementioned technical problems. The technical solution provided by this invention explores the influence of weight distribution on quantization error and uses a differential evolution algorithm to find the optimal combination of different bit widths for quantization of different layers, thereby achieving mixed-precision quantization of deep neural networks. This method has the advantages of relatively low computational complexity and strong adaptability.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a mixed-precision quantization method for deep neural networks, comprising the following steps:

[0008] Step 1: Based on the differential evolution algorithm, multiple initial quantization models of the deep neural network to be quantized with mixed precision are randomly generated, and an initialized population is obtained by using multiple initial quantization models and elite vectors as individuals; wherein, the elite vectors are the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision, and the obtained quantization bit width dictionary is determined; the values ​​of each sub-component in each individual in the initialized population represent the quantization bit width of each layer of the deep neural network;

[0009] Step 2: Use the mutation strategy in the differential evolution algorithm to diversify the population and obtain a mutated population;

[0010] Step 3: Using a binary crossover operation, the sub-components of the mutated individuals in the mutated population are recombinated with the sub-components of the target individuals in the current population to obtain multiple experimental individuals.

[0011] Step 4: Perform mixed-precision quantization on the deep neural network based on the test individuals and the target individuals to obtain the mixed-precision quantized deep neural network; use the test precision of the mixed-precision quantized deep neural network on the test set and the weight compression ratio of the mixed-precision quantized deep neural network as the fitness function of the differential evolution algorithm to select the better individual from the test individuals and the target individuals;

[0012] Step 5: Based on the selected better individuals, determine whether the differential evolution algorithm has converged. If it has converged, obtain the final mixed precision quantized deep neural network; otherwise, use the selected better individuals as the next generation population and repeat steps 2 to 5.

[0013] A further improvement of the present invention is that,

[0014] The steps to determine the quantization bit width dictionary based on the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision include:

[0015] The kurtosis and skewness of the weights of each layer in the deep neural network are calculated in layers. The kurtosis and skewness of each layer weight are mapped to a preset interval using different mapping methods. The mapped values ​​of kurtosis and skewness of each layer weight are added proportionally to obtain the quantization sensitivity weight value of each layer.

[0016] The interval between the maximum and minimum values ​​of the quantization sensitivity weight values ​​of each layer is divided into multiple intervals according to the number of quantization bit width values. Each interval after equal division corresponds to a quantization bit width value.

[0017] The quantization bit width values, represented by the intervals to which the quantization sensitivity weights of each layer belong, are assigned to the corresponding layers of the deep neural network to obtain the quantization bit width dictionary.

[0018] A further improvement of the present invention is that,

[0019] For the k-th layer of a deep neural network, the expressions for calculating, mapping, and proportionally adding kurtosis and skewness are as follows:

[0020]

[0021]

[0022] Ku′ k =((Ku k -Ku min ) / (Ku max -Ku min ))×(b-α)+a;

[0023] Sk′ k =b-(|Sk) k |-|Sk| min ) / (|Sk| max -|Sk| min )×(ba);

[0024] M k =α×Ku′ k +β×Sk′ k ;

[0025] In the formula, Ku k x represents the kurtosis of the k-th layer; i Let μ be the weight value of the i-th layer in the k-th layer, and μ be the mean weight value of the k-th layer. N is the total number of weight values ​​in the k-th layer; δ is the standard deviation of the N weight values ​​in the k-th layer; Sk k Let be the skewness of the k-th layer; a is the minimum quantization bit width that the deep neural network can be quantized to; b is the maximum quantization bit width that the deep neural network can be quantized to; Ku′ kKu represents the kurtosis value after mapping at the k-th layer. min Ku represents the minimum kurtosis value among all layers. max Sk′ represents the maximum value among the kurtosis values ​​of each layer. k The value after skewness mapping for the k-th layer; |Sk| min The minimum absolute value of the skewness values ​​for each layer; |Sk| max M is the maximum absolute value of the skewness values ​​for each layer; k Let be the quantization sensitivity weight value of the k-th layer, and α and β be the scaling factors for kurtosis and skewness, respectively.

[0026] A further improvement of the present invention is that,

[0027] The steps to obtain an initialized population using multiple initial quantization models and elite vectors as individuals include:

[0028] Assign the value of the elite vector to the first individual in the population;

[0029] Excluding the first individual, the j-th sub-component value of the d-th individual is represented as:

[0030] X d，j = a + rand(0,1) × (ba);

[0031] In the formula, X d,j is the value of the j-th subcomponent of the d-th individual; rand(0,1) represents a random real number between (0,1); a is the minimum quantization bit width that the deep neural network can quantize, and b is the maximum quantization bit width that the deep neural network can quantize.

[0032] A further improvement of the present invention is that,

[0033] Step 4 specifically includes the following steps:

[0034] Step 4.1, the obtained test individuals and target individuals The values ​​of each sub-component are used as the quantization bit width of each layer in the deep neural network. Asymmetric quantization is used to perform mixed-precision quantization on the deep neural network to obtain the experimental individual. and target individuals A deep neural network with mixed-precision quantization; where G is the number of iterations;

[0035] Step 4.2, Obtain test individuals and target individuals Weight compression ratio of deep neural networks with mixed-precision quantization and test accuracy on the test set

[0036] Step 4.3, based on the obtained weighted compression ratio and test accuracy on the test set For test individuals and target individuals A selection process is conducted, and individuals that meet the selection criteria are considered the better individuals.

[0037] A further improvement of the present invention is that,

[0038] The selection rule is as follows:

[0039] When the test accuracy on the test set corresponding to both the experimental individual and the target individual meets the requirements of the deep neural network for accuracy loss, and the weight compression ratio corresponding to the experimental individual is greater, the experimental individual is selected as the better individual.

[0040] When the test accuracy on the test set corresponding to the experimental individual meets the requirements of the deep neural network for accuracy loss, and the test accuracy on the test set corresponding to the target individual does not meet the requirements of the deep neural network for accuracy loss, the experimental individual is selected as the better individual.

[0041] When the test accuracy on the test set corresponding to both the experimental individual and the target individual does not meet the requirements of the deep neural network for accuracy loss, and the weight compression ratio corresponding to the experimental individual is greater, the experimental individual is selected as the better individual.

[0042] In other cases, the target individual is selected as the better individual.

[0043] A further improvement of the present invention is that,

[0044] In step 3, the crossover factor of the binary crossover operation is 0.9.

[0045] In a second aspect, the present invention provides a mixed-precision quantization system for deep neural networks, used to implement the mixed-precision quantization method for deep neural networks described in the first aspect of the present invention, wherein the mixed-precision quantization system for deep neural networks comprises:

[0046] The population initialization module is used to randomly generate multiple initial quantization models of the deep neural network to be quantized with mixed precision based on the differential evolution algorithm, and to obtain an initialized population using multiple initial quantization models and elite vectors as individuals; wherein, the elite vectors are the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision, and the obtained quantization bit width dictionary is determined; the values ​​of each sub-component in each individual in the initialized population represent the quantization bit width of each layer of the deep neural network;

[0047] The mutant population acquisition module is used to perform diversity processing on the population using the mutation strategy in the differential evolution algorithm to obtain a mutant population.

[0048] The test individual acquisition module is used to recombine the sub-components of mutated individuals in the mutated population with the sub-components of the target individuals in the current population using a binary crossover operation to obtain multiple test individuals.

[0049] The module for obtaining the better individual is used to perform mixed-precision quantization on the deep neural network based on the test individual and the target individual to obtain the mixed-precision quantized deep neural network; the test precision of the mixed-precision quantized deep neural network on the test set and the weight compression ratio of the mixed-precision quantized deep neural network are used as the fitness function of the differential evolution algorithm to select the better individual from the test individual and the target individual;

[0050] The convergence judgment module is used to determine whether the differential evolution algorithm has converged based on the selected better individuals. If it has converged, the final mixed precision quantized deep neural network is obtained. Otherwise, the selected better individuals are used as the next generation population, and the steps in the mutation population acquisition module, the test individual acquisition module, the better individual acquisition module, and the convergence judgment module are repeatedly executed.

[0051] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a mixed-precision quantization method for a deep neural network as described in any one of the first aspects of the present invention.

[0052] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a mixed-precision quantization method for a deep neural network as described in any one of the first aspects of the present invention.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The mixed-precision quantization method for deep neural networks provided by this invention addresses the computational complexity issue of existing mixed-precision quantization techniques based on reinforcement learning by employing Differential Evolution (DE) to search for the optimal quantization bit width for each layer of the neural network. Compared to the complex policy network training, environment modeling, and extensive exploration process of reinforcement learning, this invention only requires obtaining the accuracy of the quantization bit width corresponding to an individual in the differential evolution algorithm on the test set and the compression factor it brings to the neural network, thus reducing computational complexity to a certain extent. Furthermore, unlike existing mixed-precision quantization techniques that rely on neural network model feature information, this invention obtains the distribution information of the weights of each layer of the neural network and uses this distribution information to search for the optimal quantization bit width. Although this method utilizes the model network's feature information to some extent, the subsequent search for the optimal quantization bit width using the differential evolution algorithm allows this invention to exhibit strong adaptability to different neural network models, better explore suitable quantization configurations, and reduce the accuracy loss caused by quantization. In summary, this invention explores the impact of weight distribution on quantization error and uses the differential evolution algorithm to find the optimal combination of different bit widths for quantization of different layers, thus achieving mixed-precision quantization of deep neural networks. Compared with some existing methods such as reinforcement learning, this invention has the advantage of relatively low computational complexity. Compared with some existing mixed-precision quantization techniques that rely on the feature information of neural network models, this invention does not depend on the specific structure and type of the neural network and exhibits strong adaptability when facing different neural network models, and can be applied to various complex models with unique structures and characteristics. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a hybrid precision quantization method for deep neural networks in an embodiment of the present invention.

[0057] Figure 2 This is a flowchart illustrating the differential evolution algorithm in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram of a hybrid precision quantization system for a deep neural network in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0060] Please see Figure 1 and Figure 2 In this embodiment of the invention, a mixed-precision quantization method for deep neural networks is provided, specifically a mixed-precision quantization method for deep neural networks based on the differential evolution algorithm, and the specific steps are as follows:

[0061] Step 1: Based on the differential evolution algorithm, multiple initial quantization models of the deep neural network to be quantized with mixed precision are randomly generated, and an initialized population is obtained by using multiple initial quantization models and elite vectors as individuals; wherein, the elite vectors are the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision, and the obtained quantization bit width dictionary is determined; the values ​​of each sub-component in each individual in the initialized population represent the quantization bit width of each layer of the deep neural network;

[0062] A further exemplary scheme includes: First, based on the weight distribution of each layer of the deep neural network to be quantized with mixed precision (exemplarily, it can be a trained deep neural network model), the skewness and kurtosis of the weights of each layer of the deep neural network are obtained; Second, based on the skewness and kurtosis of the weights of each layer of the deep neural network, a better quantization bit width dictionary is determined and used as the elite vector; Then, multiple initial quantization models are added to the elite vector to form an initialized population corresponding to the deep neural network.

[0063] Step 2: Use the mutation strategy in the differential evolution algorithm to diversify the population and obtain a mutated population;

[0064] Step 3: Use a binary crossover operation to recombine the mutated individuals in the mutated population with the sub-components of the target individuals in the current population to obtain multiple experimental individuals; in a specific exemplary scheme, the crossover factor of the binary crossover operation is 0.9.

[0065] Step 4: Perform mixed-precision quantization on the deep neural network based on the test individuals and the target individuals to obtain the mixed-precision quantized deep neural network; use the test precision of the mixed-precision quantized deep neural network on the test set and the weight compression ratio of the mixed-precision quantized deep neural network as the fitness function of the differential evolution algorithm to select the better individual from the test individuals and the target individuals;

[0066] Step 5: Based on the better individuals obtained by selection, determine whether the differential evolution algorithm has converged. If it has converged, obtain the final deep neural network after mixed precision quantization. Otherwise, use the better individuals obtained by selection as the next generation population and repeat steps 2 to 5.

[0067] The technical solution provided in this invention explores the impact of weight distribution on quantization error and uses the differential evolution algorithm to find the optimal combination of different bit widths for quantization of different layers, thereby achieving mixed-precision quantization of deep neural networks. Compared with some existing methods such as reinforcement learning, this invention only requires iterating the results of the differential evolution algorithm until convergence, without the need for repeated trial and error and learning processes like reinforcement learning, thus reducing computational complexity. Compared with some existing mixed-precision quantization techniques that rely on the feature information of neural network models, this invention does not require special adjustments and optimizations for each deep neural network, and is not dependent on the specific structure and type of the neural network. When facing different neural network models, mixed-precision quantization can be achieved by exploring weight distribution and using the differential evolution algorithm, which has wider adaptability and can be applied to various complex models with unique structures and characteristics.

[0068] In one embodiment of the present invention, step 1, which involves randomly generating multiple initial quantization models of the deep neural network to be quantized with mixed precision based on the differential evolution algorithm, and using the multiple initial quantization models and elite vectors as individuals to obtain an initialized population, specifically includes:

[0069] The deep neural network to be mixed-precision quantization is an N f A deep neural network F with multiple layers;

[0070] The steps to determine the quantization bit width dictionary based on the skewness and kurtosis of the weights of each layer of the deep neural network F are as follows: calculate the kurtosis and skewness of the weights of each layer of the deep neural network F layer by layer, map the kurtosis and skewness of each layer weight to the preset interval [a,b] using different mapping methods, and add the mapped values ​​of kurtosis and skewness proportionally to obtain the quantization sensitivity weight value of each layer.

[0071] The interval between the maximum and minimum values ​​of the quantization sensitivity weight values ​​of each layer is divided into multiple intervals according to the number of quantization bit width values, and each interval after equal division corresponds to a quantization bit width value.

[0072] The quantization bit width values ​​represented by the intervals to which the quantization sensitivity weight values ​​of each layer belong are assigned to the corresponding layers of the deep neural network to obtain the quantization bit width dictionary;

[0073] Specifically, for the k-th layer of a deep neural network F, the calculation expression is as follows:

[0074]

[0075]

[0076] Ku′ k =((Ku k -Ku min ) / (Ku max -Ku min ))×(ba)+a;

[0077] Sk′ k =b-(|Sk) k |-|Sk| min ) / (|Sk| max -|Sk| min )×(b-α);

[0078] M k =α×Ku′ k +β×Sk′ k ;

[0079] In the formula, Ku k x represents the kurtosis of the k-th layer; i Let μ be the weight value of the i-th layer in the k-th layer, and μ be the mean weight value of the k-th layer. N is the total number of weight values ​​in the k-th layer; δ is the standard deviation of the N weight values ​​in the k-th layer; Sk k Let be the skewness of the k-th layer; a is the minimum quantization bit width that the deep neural network F can be quantized to; b is the maximum quantization bit width that the deep neural network F can be quantized to; Ku′ k Ku represents the kurtosis value after mapping at the k-th layer. min Ku represents the minimum kurtosis value among all layers. max Sk′ represents the maximum value among the kurtosis values ​​of each layer. k The value after skewness mapping for the k-th layer; |Sk| min The minimum absolute value of the skewness values ​​for each layer; |Sk| max M is the maximum absolute value of the skewness values ​​for each layer; kα represents the quantization sensitivity weight value of the k-th layer, and α and β are the proportional coefficients of kurtosis and skewness, respectively, used to balance the influence of kurtosis and skewness.

[0080] In the step of obtaining the initialized population using multiple initial quantization models and elite vectors as individuals,

[0081] The initial population size NP is set to 100, the individual dimension D is equal to the number of layers in the deep neural network, the upper bound of the search space is b (interpretably, the maximum quantization bit width is b bits; in the exemplary scheme, the value of b can be 8), and the lower bound of the search space is a (interpretably, the minimum quantization bit width is a bits; in the exemplary scheme, the value of a can be 2). The elite vector value is assigned to the first individual, and the values ​​of the remaining individuals are randomly generated. For the d-th individual, the value of the j-th dimension is randomly generated between a and b. Then, each sub-component value is rounded to ensure that each sub-component value of an individual is an integer, represented as:

[0082] X d，j = a + rand(0, 1) × (ba);

[0083] In the formula, X d，j Let be the value of the j-th sub-component of the d-th individual; rand(0,1) represents a random real number between (0,1);

[0084] For each pair of d and j, rand(0,1) will generate new values ​​that are independent of each other; after initialization, the population will perform mutation, crossover, and selection operations in sequence until the algorithm meets the termination condition.

[0085] In one embodiment of the present invention, step 4 specifically includes:

[0086] The obtained test individuals and target individuals The values ​​of each sub-component are used as the quantization bit width of each layer in the deep neural network. Asymmetric quantization is used to perform mixed-precision quantization on the deep neural network to obtain the experimental individual. and target individuals A deep neural network with mixed-precision quantization; where G is the number of iterations;

[0087] Obtaining test individuals and target individuals Weight compression ratio of deep neural networks with mixed-precision quantization and test accuracy on the test set

[0088] The experimental individuals were evaluated based on the obtained weighted compression ratio and the test accuracy on the test set. and target individuals Select individuals with both superior weight compression ratio and test accuracy on the test set to enter the next generation of the population.

[0089] For the d-th individual The selection rules are expressed as follows:

[0090] when and and hour,

[0091] When the test accuracy on the test set corresponding to both the experimental individual and the target individual meets the requirements of deep neural networks for accuracy loss, and the weight compression ratio corresponding to the experimental individual is greater, the experimental individual is selected to enter the next generation.

[0092] when and hour,

[0093] When the test accuracy on the test set corresponding to the experimental individual meets the requirements of the deep neural network for accuracy loss, but the test accuracy on the test set corresponding to the target individual does not meet the requirements of the deep neural network for accuracy loss, the experimental individual is selected to enter the next generation.

[0094] when and and hour,

[0095] When the test accuracy on the test set corresponding to both the experimental individual and the target individual does not meet the accuracy loss requirements of the deep neural network, but the weight compression ratio of the experimental individual is greater, the experimental individual is selected to enter the next generation.

[0096] In other cases, select the target individual to enter the next generation.

[0097] In the formula, acc F It is the original accuracy of the trained deep neural network on the test set before quantization, Δ acc This is the maximum acceptable loss of accuracy. The individual selected is the representative of the next generation of the population.

[0098] In summary, this invention discloses a hybrid precision quantization scheme for deep neural networks based on differential evolution. It utilizes the distribution information of the weights at each layer of the deep neural network to obtain a superior individual (elite vector) with quantization bit width information at each layer. Multiple initial quantization models are randomly generated for the deep neural network. An initialized population is obtained based on the elite vector and the combination of these initial quantization models. Mutation and crossover operators are used to perform diversity processing on the population. The test precision of the quantized deep neural network model on the test set and the weight compression ratio of the quantized deep neural network model after compression are used together as the objective function of the differential evolution algorithm to find a better individual to guide the quantization of the neural network. The method described in this invention reduces the computational and storage requirements of the network model while improving the model's operating efficiency in resource-constrained environments such as embedded devices, enabling edge intelligence and mobile applications.

[0099] The principle explanation for the feasibility of the technical solution of this invention is as follows: By analyzing the quantization-related properties of different layers in a neural network, it was found that in a deep neural network, different layers have different structures and different quantization-related properties. Therefore, performing the same quantization on all layers is not the optimal result. Trying to set different bit widths for different layers can obtain better results. The technical solution of this invention utilizes the distribution of weights in each layer of the neural network and the differential evolution algorithm to search for the optimal quantization bit width of each layer of the neural network. This allows the neural network model to be significantly compressed while meeting the accuracy requirements of the test set.

[0100] In one specific embodiment of the present invention, based on the Windows operating system and the Pytoch framework, deep neural networks such as LeNet5 and ResNet20 are used to test the MNIST dataset and the CIFAR-10 dataset, respectively. In this embodiment of the present invention, the quantization bits used for weights are abbreviated as "w bits", the test accuracy is abbreviated as "Acc", and the compression ratio of the weights is abbreviated as "W-Comp". The experimental results are shown in Tables 1 and 2.

[0101] Table 1. Quantization results of LeNet-5 on MNIST and the quantization bit width of each layer

[0102]

[0103]

[0104] In this embodiment of the invention, the accuracy of the original LeNet-5 model provided by PyTorch on the MNIST dataset reached 99.14%. As can be seen from Table 1, the neural network quantization (DEQ) method based on differential evolution algorithm proposed in this invention achieves good compression effect, with a compression ratio of 11.53× for the quantization of neural network weights, and the accuracy only decreased by 0.05%. This shows that the quantization method proposed in this invention is effective and achieves a relatively high compression ratio with extremely low accuracy loss.

[0105] Table 2. Comparison of quantization results of ResNet20 on Cifar-10

[0106] DEQ 10.31× 2.63 4-bit integer quantization 8.00× 3.83

[0107] In this embodiment of the invention, the accuracy of the original ResNet20 model provided by PyTorch on the Cifar-10 dataset reached 92.02%. As can be seen from Table 2, the Neural Network Quantization (DEQ) method based on the Differential Evolution algorithm proposed in this invention achieves good compression effect on the ResNet20 model, with a compression ratio of 10.31× for the quantization of neural network weights and an accuracy decrease of only 2.63%. Compared with the 4-bit integer quantization method, the method of this embodiment is more effective and achieves a greater compression ratio and model accuracy.

[0108] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0109] Please see Figure 3 In this embodiment of the invention, a mixed-precision quantization system for deep neural networks is provided, used to implement the mixed-precision quantization method for deep neural networks described above in this embodiment of the invention. The mixed-precision quantization system for deep neural networks includes:

[0110] The population initialization module is used to randomly generate multiple initial quantization models of the deep neural network to be quantized with mixed precision based on the differential evolution algorithm, and to obtain an initialized population using multiple initial quantization models and elite vectors as individuals; wherein, the elite vectors are the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision, and the obtained quantization bit width dictionary is determined; the values ​​of each sub-component in each individual in the initialized population represent the quantization bit width of each layer of the deep neural network;

[0111] The mutant population acquisition module is used to perform diversity processing on the population using the mutation strategy in the differential evolution algorithm to obtain a mutant population.

[0112] The test individual acquisition module is used to recombine the sub-components of mutated individuals in the mutated population with the sub-components of the target individuals in the current population using a binary crossover operation to obtain multiple test individuals.

[0113] The module for obtaining the better individual is used to perform mixed-precision quantization on the deep neural network based on the test individual and the target individual to obtain the mixed-precision quantized deep neural network; the test precision of the mixed-precision quantized deep neural network on the test set and the weight compression ratio of the mixed-precision quantized deep neural network are used as the fitness function of the differential evolution algorithm to select the better individual from the test individual and the target individual;

[0114] The convergence judgment module is used to determine whether the differential evolution algorithm has converged based on the selected better individuals. If it has converged, the final mixed precision quantized deep neural network is obtained. Otherwise, the selected better individuals are used as the next generation population, and the steps in the mutation population acquisition module, the test individual acquisition module, the better individual acquisition module, and the convergence judgment module are repeatedly executed.

[0115] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to execute the operation of a mixed-precision quantization method for deep neural networks.

[0116] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the mixed-precision quantization method for deep neural networks in the above embodiments.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A hybrid precision quantization method for deep neural networks, characterized in that, Includes the following steps: Step 1: Based on the differential evolution algorithm, multiple initial quantization models of the deep neural network to be quantized with mixed precision are randomly generated, and an initialized population is obtained by using multiple initial quantization models and elite vectors as individuals; wherein, the elite vectors are the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision, and the obtained quantization bit width dictionary is determined; the values ​​of each sub-component in each individual in the initialized population represent the quantization bit width of each layer of the deep neural network; Step 2: Use the mutation strategy in the differential evolution algorithm to diversify the population and obtain a mutated population; Step 3: Using a binary crossover operation, the sub-components of the mutated individuals in the mutated population are recombinated with the sub-components of the target individuals in the current population to obtain multiple experimental individuals. Step 4: Perform mixed-precision quantization on the deep neural network based on the test individuals and the target individuals to obtain the mixed-precision quantized deep neural network; use the test precision of the mixed-precision quantized deep neural network on the test set and the weight compression ratio of the mixed-precision quantized deep neural network as the fitness function of the differential evolution algorithm to select the better individual from the test individuals and the target individuals; wherein, the deep neural network is a convolutional deep neural network used to perform image recognition; the test set is a selected image sample dataset; Step 5: Based on the selected better individuals, determine whether the differential evolution algorithm has converged. If it has converged, obtain the final mixed precision quantized deep neural network; otherwise, use the selected better individuals as the next generation population and repeat steps 2 to 5. The steps for determining the quantization bit width dictionary based on the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision include: The kurtosis and skewness of the weights of each layer in the deep neural network are calculated in layers. The kurtosis and skewness of each layer weight are mapped to a preset interval using different mapping methods. The mapped values ​​of kurtosis and skewness of each layer weight are added proportionally to obtain the quantization sensitivity weight value of each layer. The interval between the maximum and minimum values ​​of the quantization sensitivity weight values ​​of each layer is divided into multiple intervals according to the number of quantization bit width values. Each interval after equal division corresponds to a quantization bit width value. The quantization bit width values, represented by the intervals to which the quantization sensitivity weights of each layer belong, are assigned to the corresponding layers of the deep neural network to obtain the quantization bit width dictionary.

2. The hybrid precision quantization method for deep neural networks according to claim 1, characterized in that, For the first deep neural network k The expressions for calculating, mapping, and proportionally adding kurtosis and skewness are as follows: ； ； ； ； ； In the formula, For the first k Kudo of the layer; For the first k Layer i Each weight value For the first k The average weight of the layer. , For the first k The total number of layer weight values; For the first k Layer The standard deviation of each weight value; For the first k Layer skewness; a It is the smallest quantization bit width that a deep neural network can be quantized to. b It is the maximum quantization bit width that a deep neural network can be quantized to; For the first k The kurtosis value after layer mapping; It is the minimum value among the kurtosis values ​​of each layer; It is the maximum value among the kurtosis values ​​of each layer; For the first k The skewness mapping value of the layer; It is the minimum value among the absolute values ​​of the skewness values ​​of each layer; It is the maximum absolute value among the skewness values ​​of each layer; For the first k The quantization sensitivity weight value of the layer and These are the scaling factors for kurtosis and skewness, respectively.

3. The hybrid precision quantization method for deep neural networks according to claim 1, characterized in that, The steps to obtain an initialized population using multiple initial quantization models and elite vectors as individuals include: Assign the value of the elite vector to the first individual in the population; Excluding the first individual, the... d The first individual j The individual component values ​​are represented as follows: ； In the formula, For the first d The first individual j Individual component values; Represents a random real number between (0, 1); a It is the smallest quantization bit width that a deep neural network can be quantized to. b It is the maximum quantization bit width that a deep neural network can be quantized to.

4. The hybrid precision quantization method for deep neural networks according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1, the obtained test individuals and target individuals The values ​​of each sub-component are used as the quantization bit width of each layer in the deep neural network. Asymmetric quantization is used to perform mixed-precision quantization on the deep neural network to obtain the experimental individual. and target individuals Deep neural networks with mixed-precision quantization; where, G This represents the number of iterations. Step 4.2, Obtain test individuals and target individuals Weight compression ratio of deep neural networks with mixed-precision quantization , and test accuracy on the test set , ; Step 4.3, based on the obtained weighted compression ratio , and test accuracy on the test set , For test individuals and target individuals A selection process is conducted, and individuals that meet the selection criteria are considered the better individuals.

5. The hybrid precision quantization method for deep neural networks according to claim 4, characterized in that, The selection rule is as follows: When the test accuracy on the test set corresponding to both the experimental individual and the target individual meets the requirements of the deep neural network for accuracy loss, and the weight compression ratio corresponding to the experimental individual is greater, the experimental individual is selected as the better individual. When the test accuracy on the test set corresponding to the experimental individual meets the requirements of the deep neural network for accuracy loss, and the test accuracy on the test set corresponding to the target individual does not meet the requirements of the deep neural network for accuracy loss, the experimental individual is selected as the better individual. When the test accuracy on the test set corresponding to both the experimental individual and the target individual does not meet the requirements of the deep neural network for accuracy loss, and the weight compression ratio corresponding to the experimental individual is greater, the experimental individual is selected as the better individual. In other cases, the target individual is selected as the better individual.

6. The hybrid precision quantization method for deep neural networks according to claim 1, characterized in that, In step 3, the crossover factor of the binary crossover operation is 0.

9.

7. A hybrid precision quantization system for deep neural networks, characterized in that, For implementing the mixed-precision quantization method for the deep neural network according to claim 1, the mixed-precision quantization system for the deep neural network includes: The population initialization module is used to randomly generate multiple initial quantization models of the deep neural network to be quantized with mixed precision based on the differential evolution algorithm, and to obtain an initialized population using multiple initial quantization models and elite vectors as individuals; wherein, the elite vectors are the skewness and kurtosis of the weights of each layer of the deep neural network to be quantized with mixed precision, and the obtained quantization bit width dictionary is determined; the values ​​of each sub-component in each individual in the initialized population represent the quantization bit width of each layer of the deep neural network; The mutant population acquisition module is used to perform diversity processing on the population using the mutation strategy in the differential evolution algorithm to obtain a mutant population. The test individual acquisition module is used to recombine the sub-components of mutated individuals in the mutated population with the sub-components of the target individuals in the current population using a binary crossover operation to obtain multiple test individuals. The module for obtaining the better individual is used to perform mixed-precision quantization on the deep neural network based on the test individual and the target individual to obtain the mixed-precision quantized deep neural network; the test precision of the mixed-precision quantized deep neural network on the test set and the weight compression ratio of the mixed-precision quantized deep neural network are used as the fitness function of the differential evolution algorithm to select the better individual from the test individual and the target individual; The convergence judgment module is used to determine whether the differential evolution algorithm has converged based on the selected better individuals. If it has converged, the final mixed precision quantized deep neural network is obtained. Otherwise, the selected better individuals are used as the next generation population, and the steps in the mutation population acquisition module, the test individual acquisition module, the better individual acquisition module, and the convergence judgment module are repeatedly executed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the mixed-precision quantization method for deep neural networks as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the mixed-precision quantization method for deep neural networks as described in any one of claims 1 to 6.

Images