A Model Compression Method Combining Bit Quantization and Bit Weight Search

Abstract

The present invention discloses a model compression method combining bit quantization and bit weight search, comprising the following steps: S1, loading image data and preprocessing the image data; S2, establishing a convolutional neural network model, and setting corresponding bit search space and weight search space in the convolutional neural network model; S3, performing joint search training on the convolutional neural network model; S4, selecting the maximum probability bit width in the bit search results to construct a lightweight network, and saving the weight parameters obtained with the maximum probability bit width during the last joint search as the weight parameters of the lightweight network, and then performing weight search to further optimize the lightweight network model, and outputting the compressed and optimized lightweight network model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and the field of lightweight compression therein, and in particular, to a model compression method combining bit quantization and bit weight search. Background Art

[0002] Deep learning has experienced rapid development and has been widely applied in many real-world scenarios, such as robotics, autonomous driving, and face recognition technology, etc. In real-world scenarios, computing devices are often resource-constrained platforms (such as storage, computing speed, etc.), so it is very necessary to study network structure compression schemes in deep learning. The key to the research and deployment of deep neural networks lies in achieving the best performance on target hardware resources.

[0003] For the model compression of deep learning, there are compression methods based on different ideas, such as network pruning to reduce unimportant branches; parameter quantization, by using fewer bits to store parameters, thereby reducing the total storage amount of parameters and achieving the compression effect; network structure redesign, such as using depthwise separable convolution to reduce the number of operations and thus achieving the effect of compressing the model. Among the above compression methods, parameter quantization has the greatest degree of model compression. After quantizing the parameters into bits, it is more adaptable to embedded systems for bit operations, thereby making the model more adaptable to the hardware and improving the model accuracy. Therefore, the present invention adopts the method of parameter quantization for model compression.

[0004] The redundancy levels of different layers in a network are different. Selecting a mixed-precision quantization method with different quantization bits for different layers will result in better performance of the compressed network. However, there are numerous combinations of quantization bits for different layers, making it difficult to determine through manual selection. Automatic search for quantization bits is a more effective solution. Considering that the existing search algorithms have a high cost and the search objects mainly focus on the network structure, the search algorithm DARTS with a relatively small cost is selected and modified into an automatic search algorithm for quantization bits. The performance of the quantization model is closely related to the quantization function. Therefore, the quantization function needs to be considered when searching for quantization bits. Most of the current parameter quantization algorithms use fixed quantization functions for model quantization, and the quantization functions suitable for different parameter distributions and different bit settings are not the same. For example, the SLB algorithm in Searching for Low-Bit Weights in Quantized Neural Networks. The above two methods, the automatic search algorithm for quantization bits and the learnable quantization function, can compress the model to a certain extent, but the compression degree and the accuracy of the resulting model will have significant limitations when used alone and cannot achieve good results. Simply cascading the above two methods, that is, first using bit search to obtain the architecture and then using weight search to optimize the weights, this method has a simple idea but has problems and cannot obtain the optimal result. Summary of the Invention

[0005] The purpose of the present invention is to provide a model compression method and its system that combines bit quantization and bit weight search to solve the above technical problems. The present invention uses weight value search to obtain a learnable quantization function, combines weight value search and quantization bit search, and makes them match to obtain better performance of the quantization model.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] A model compression method that combines bit quantization and bit weight search includes the following steps:

[0008] S1. Load image data and preprocess the image data;

[0009] S2. Establish a convolutional neural network model, and set the corresponding bit search space and weight search space in the convolutional neural network model;

[0010] S3. Conduct joint search training on the convolutional neural network model;

[0011] S4. Select the maximum probability bit width in the bit search result to construct a lightweight network, save the weight parameters obtained with the maximum probability bit width at the end of the joint search as the weight parameters of the lightweight network, and then conduct weight search to optimize the lightweight network model, and output the compressed and optimized lightweight network model.

[0012] Further, the preprocessing includes padding, cropping, and image enhancement of the image data.

[0013] Further, the loaded image data is cut into a training set, a validation set, and a test set.

[0014] Further, a gradient search strategy is used to search and train the convolutional neural network model. The bit limit and bit weight values of the convolutional neural network model are alternately searched in the training set and the validation set, and the bits and weights are selected to be the overall best.

[0015] Further, in the joint search, for each selection of bits and weights obtained in the search, class softmax is used for normalization to achieve continuous relaxation of the selection of bits and weights. Furthermore, gradient descent can be used for optimization search during training, and the temperature exponential decay is set to make the probability matrix of the selected bits and weights approximate one-hot encoding, so that the selection of bits and weights can converge to a single selection.

[0016] Further, for one layer of the convolutional neural network model, assuming there are a total of N selectable bit numbers, the selection probability of the m-th bit is calculated by the following formula:

[0017]

[0018] where p s represents the bit selection probability, and further represents the selection probability of the m-th bit, N is the number of types of selectable bit numbers, is the selection weight before normalization of the m-th bit number, τ0 is the set decay temperature, is the selection weight before normalization of the j-th bit number.

[0019] Further, for one layer of the convolutional neural network model, assuming there are a total of N selectable bit numbers, and assuming that the m-th bit number corresponds to v m selectable quantization weight values, the selection probability of the i-th quantization weight value of the m-th bit number in the search is calculated by the following formula:

[0020]

[0021] where p represents the selection probability of the quantization weight value, and further with added superscripts and subscripts represents the selection probability of the i-th quantization weight value of the m-th bit number in the search; v m represents the number of selectable quantization weight values corresponding to the m-th bit number; is a learnable parameter, which is the selection weight before normalization of the j-th quantization weight value of the m-th bit. τ1 is a preset decay temperature. is the selection weight before normalization of the i-th quantization weight value of the m-th bit.

[0022] Furthermore, the output function during the training of the convolutional neural network model is:

[0023] Y = σ(W * X + b) (3)

[0024] where Y represents the output during model training, σ(·) is the activation function, X is the input of the convolutional layer, b is the bias vector of the convolutional layer, and W is the convolutional weight during training, and its elements are:

[0025]

[0026] where represents the selection probability of searching for the m-th bit; N is the number of types of selectable bits; is the continuous weight element under the m-th quantization bit, and it corresponds to v m selectable quantization weight values, which are calculated by the following formula:

[0027]

[0028] V = {v1, v2,..., v2q} (6)

[0029] where is the selection probability of the i-th quantization weight value of the m-th bit during the search, V is the set of quantization values for bit weight search, and for the set V of quantization values for bit weight search with q bits, there are 2 q discrete weight values. Therefore, the v i is the i-th normalized bit weight quantization value, and v 2q is the 2 q th discrete weight value.

[0030] Furthermore, the output function during the inference of the convolutional neural network model is:

[0031] Y = σ(W q * X + b) (7)

[0032] where Y represents the output during model training, σ(·) is the activation function, X is the input of the convolutional layer, b is the bias vector of the convolutional layer, and W q is the convolutional weight during training, and its elements are:

[0033]

[0034] where is the discrete bit weight at the m-th quantization bit, corresponding to v m available quantization weight values, and its calculation formula is as follows:

[0035]

[0036]

[0037] where I m represents the position corresponding to the maximum selection probability value of the quantization weight value of the m-th bit number in the search, that is, a vector of length v m vector, where the element corresponding to the position of the maximum selection weight value is 1, and the rest are 0; represents the value of the j-th position of the one-hot vector I m ; v j represents the j-th normalized bit weight quantization value in the set V of bit weight search quantization values; is the selection probability of the i-th quantization weight value of the m-th bit number in the search.

[0038] J represents the one-hot vector corresponding to the position of the maximum selection probability of the bit number in the search, and its calculation formula is

[0039]

[0040] Since there are a total of N available bit numbers, J is a vector of length N, where the element corresponding to the position of the maximum selection weight value is 1, and the rest are 0. J (m) represents the value of the m-th position of the one-hot vector J, represents the selection probability of the i-th bit in the search.

[0041] Furthermore, during the search training, a loss term for computing resources and a loss term for storage resources are added to the corresponding loss function to simultaneously optimize the computing cost and storage cost. The obtained loss function is:

[0042] L = L c + β compute ·L compute + γ memory ·L memory (12)

[0043] where L c is the cross-entropy loss of the convolutional neural network model, L compute is the number of bit operations used by the convolutional neural network model, β compute is the adjustable weight corresponding to the total computing cost, L memory is the total storage amount involved in the convolutional neural network model, γ memoryIs the adjustable weight corresponding to the total storage capacity.

[0044] Compared with the existing technology, the beneficial effects of the present invention are as follows:

[0045] According to the provided hardware resource constraints and the expected model accuracy, the present invention uses a neural network architecture to search for the number of bit quantizations and the number of bit weights, and then obtains an efficient lightweight convolutional neural network model suitable for the provided hardware constraints; using the idea of class softmax, the selected weights are normalized and relaxed, and an appropriate temperature is set to make the selected probability matrix of the obtained bits and inference weights approximate one-hot encoding, making the selection probability concentrated. During training, full-precision weights are used to reduce the error caused by quantization; using the idea of gradient descent, the discretized selection space can be efficiently searched, and the most ideal model parameters can be reached faster, thereby saving a large amount of computing resources and search time. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Is a schematic flowchart of the model compression method described in the present invention;

[0047] Figure 2 Is a schematic diagram of the joint search process of the model compression method described in the present invention;

[0048] Figure 3 Is a schematic diagram of the network inference process of the model compression method described in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] In order to enable those skilled in the art to better understand the solution of the present invention, the following will further describe the present invention in detail with reference to the drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0050] Using bit architecture search, a convolutional neural network architecture that meets the hardware constraints can be obtained; while using discrete weight search, a bit convolutional neural network model can be optimized maximally. The goal of the present invention is to obtain the optimal lightweight network model. Simply cascading the above two methods, that is, using bit search to obtain the bit network architecture and then using weight search to optimize its weights, this idea is simple but has problems and cannot obtain the optimal result. Therefore, the present invention adopts a joint search method for both, performing bit search and weight search simultaneously, which can not only meet the existing hardware resources, but also achieve the ideal accuracy and compression rate, combining the advantages of both methods.

[0051] Embodiment 1

[0052] The model compression method combining bit quantization and bit weight search in this embodiment is specifically as follows:

[0053] S1. Load the original image, cut the original image dataset into a training set, a validation set, and a test set, and perform a series of preprocessing operations on the dataset, such as padding, cropping, and image enhancement. The training set and the validation set are used for searching and optimizing the convolutional neural network model in step S3.

[0054] S2. According to the hardware environment, the required model accuracy, and complexity, establish a convolutional neural network model. The establishment of this convolutional neural network model can be based on engineering experience or refer to existing lightweight networks. After the convolutional neural network model is established, the corresponding bit search space and weight search space in the convolutional neural network model need to be set. The bit search space is the number of bits of a specific convolutional layer, and the weight search space is the discrete bit weight under specific bit constraints. Bit search aims to search for the weight bit width of the convolutional layer, while weight search aims to search for the weight value under fixed weight bits.

[0055] S3. By building Darts_SLB_Conv (a convolutional layer combining bit quantization and bit weight search) and using the gradient search strategy to perform neural network search training, alternately search for the bit constraints and bit weight values of the model in the training set and the validation set. In addition, step S3 jointly searches for the selection space of bits and weights.

[0056] Specifically:

[0057] In the joint search, for each selection of bits and weights obtained in the search, use class softmax for normalization, continuously relax it, and then gradient descent can be used for optimization search during training. At the same time, set the temperature exponential decay to make the obtained bit and weight selection probability matrix approximate one-hot encoding, so that it can converge to a single selection.

[0058] Specifically, the sum of the selection vector probabilities of each group is 1, and set the temperature τ to decay exponentially from a relatively large number to a small value. For example, decaying from 10 to 0.01 exponentially will have a better effect, making the obtained bit and weight selection probability matrix approximate one-hot encoding. Let the transformation tensor be The core parameter of bit search is It is indicated that the core parameter N for weight search is the number of choices of quantization bits.

[0059] For one layer of the convolutional neural network model, assuming there are a total of N selectable numbers of bits, then the selection probability of searching for the mth bit is calculated by the following formula:

[0060]

[0061] Among them, p s represents the bit selection probability, and further represents the selection probability of searching for the m-th bit. N is the number of types of selectable bits, is a learnable parameter, which is the selection weight before normalization of the m-th bit number, and τ0 is a preset decay temperature.

[0062] Furthermore, for one layer of the convolutional neural network model, assume that there are a total of N types of selectable bit numbers, and assume that the m-th bit number corresponds to v m selectable quantization weight values. Then, the selection probability of the i-th quantization weight value of the m-th bit number in the search is calculated by the following formula:

[0063]

[0064] Among them, p represents the selection probability of the quantization weight value, and further with added superscripts and subscripts represents the selection probability of the i-th quantization weight value of the m-th bit number in the search; v m represents the number of selectable quantization weight values corresponding to the m-th bit number; is a learnable parameter, which is the selection weight before normalization of the j-th quantization weight value of the m-th bit number, and τ1 is a preset decay temperature.

[0065] The purpose of bit quantization search is to search for an optimal discrete bit quantization network in the search space. Let the input of this layer be X, the convolutional weight be W b , and the bias vector be b. Then, the convolutional operation under the i-th quantization bit of the convolutional layer is expressed as W b *X + b; the number of bit quantization selections is N; for the probabilities of each bit quantization selection, perform the above probability normalization calculation to obtain the probability of the m-th bit weight selection Therefore, its output Y is as follows:

[0066] Y = σ(W b *X + b) (15)

[0067] Among them, σ(·) is the activation function; W b is the convolutional weight, and the calculation formula for its elements is:

[0068]

[0069] Among them, represents the selection probability of searching for the m-th bit; N is the number of types of selectable bits; is the weight of the m-th quantization bit, and it corresponds to v mA set of selectable quantization weight values.

[0070] The purpose of bit weight search is to optimize the discrete weight values under a fixed bit width and improve the network accuracy. Let \(V = \{v_1, v_2, \ldots, v_{2^q}\}\) be the set of quantization values for bit weight search. For \(q\)-bit \(V\), there are \(2^q\) discrete weight values. The \(x\)-th element \(v_x\in[0,1]\) in set \(V\) is the normalized bit weight quantization value. Suppose the \(m\)-th bit number corresponds to \(v_m\) selectable quantization weight values. Then the continuous weight elements under the \(m\)-th quantization bit are calculated as follows: 2q} is the set of quantization values for bit weight search. For \(q\)-bit \(V\), there are \(2^q\) q discrete weight values. The \(x\)-th element v x ∈[0,1] in set \(V\) is the normalized bit weight quantization value. Suppose the \(m\)-th bit number corresponds to \(v_m\) m selectable quantization weight values. Then the continuous weight elements under the \(m\)-th quantization bit are calculated as follows:

[0071]

[0072] The output function of the network optimized by joint bit and weight search is different during network training and network inference. During network training and inference, the quantization bit search methods are the same, both using weight normalization for relaxation. In terms of quantization weights, training uses the relaxed continuous weights, while inference uses the discrete bit quantization weights.

[0073] Combining Equation (16) and Equation (17), that is, combining the network training in bit quantization search and bit weight search, the output is as follows:

[0074] Y = σ(W * X + b) (18)

[0075] where \(Y\) represents the output during model training, σ(·) is the activation function, \(X\) is the input of the convolutional layer, \(b\) is the bias vector of the convolutional layer, and \(W\) is the convolutional weight during training, and its elements are:

[0076]

[0077] Furthermore, the output function of network inference is derived, and its formula is as follows:

[0078] Y = σ(W q * X + b) (20)

[0079] where \(Y\) represents the output during model training, σ(·) is the activation function, \(X\) is the input of the convolutional layer, \(b\) is the bias vector of the convolutional layer, and \(W q is the convolutional weight during training, and its elements are:

[0080]

[0081] where, is the discrete bit weight under the \(m\)-th quantization bit, which corresponds to \(v_m\) mThere are several selectable quantization weight values, and their calculation formulas are as follows:

[0082]

[0083]

[0084] Among them, I m represents the position corresponding to the maximum selection probability value of the quantization weight value of the m-th bit in the search, that is, a one-hot vector with a length of v m vector, where the element corresponding to the position of the maximum selection weight is 1, and the rest are 0; represents the value of the j-th position of the one-hot vector I m ; v j represents the j-th normalized bit weight quantization value in the set V of bit weight search quantization values; is the selection probability of the i-th quantization weight value of the m-th bit in the search.

[0085] J represents the one-hot vector corresponding to the position of the maximum selection probability of the number of bits in the search, and its calculation formula is

[0086]

[0087] Since there are a total of N selectable numbers of bits, J is a vector with a length of N, where the element corresponding to the position of the maximum selection weight is 1, and the rest are 0. J (m) represents the value of the m-th position of the one-hot vector J, represents the selection probability of the i-th bit in the search.

[0088] Determine the loss function. Since the resource constraints of different hardware platforms are different, the network structures and parameters to be searched will also be different. Therefore, in order to meet the corresponding hardware constraints during the search, relevant losses of the computing cost and storage cost need to be added to the loss function. The computing cost of a single network is described based on the number of floating-point operations and the number of fixed-point bit operations in the network, and then the total computing cost of all networks in the entire selection space is calculated as L compute , and β compute is used as the adjustable weight of the computing cost; at the same time, we also introduce the relevant loss of the storage cost, and the storage cost L of the network is described based on the storage amount of the fixed parameters of the network and the data cache introduced by the calculation during the operation of the network memory , and γ memory is also used as the adjustable weight. Therefore, the newly added loss function is:

[0089] L = L c + β compute ·Lcompute +γ memory ·L memory (25)

[0090] where L c is the cross-entropy loss of the model.

[0091] S4. After the joint search ends, select the maximum probability bit width in the bit search results, construct a new lightweight network based on this, and save the bit weight parameters obtained finally during the joint search as the weight parameters of the new network. Then perform bit weight search to further optimize the lightweight network model, and finally output the compressed and optimized lightweight network model.

[0092] For the model compression method combining bit quantization and bit weight search according to the present invention, the construction and optimization of the lightweight network are carried out by using the specific steps and methods described above.

[0093] Embodiment 2

[0094] In this example, the loss function uses

[0095] L = L c +β compute ·L compute

[0096] where, L c is the cross-entropy loss of the model, L compute is the total sum of the computational costs of all networks under the entire selection space, and β compute is the adjustable weight for the computational cost.

[0097] Embodiment 3

[0098] In this example, the loss function uses:

[0099] L = L c +γ memory ·L memory

[0100] where, L c is the cross-entropy loss of the model, L memory describes the storage cost of the network based on the storage amount of the fixed parameters of the network and the cache of data introduced during the network operation, and γ memory is used as the adjustable weight for the storage cost.

[0101] For those skilled in the art, various corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A model compression method combining bit quantization and bit weight search, characterized in that It includes the following steps: S1. Load the image data and preprocess the image data; S2. Establish a convolutional neural network model and set the corresponding bit search space and weight search space in the convolutional neural network model; S3. Conduct joint search training on the convolutional neural network model; for each selection of bits and weights obtained in the search, use the class softmax for normalization to achieve continuous relaxation of the selection of bits and weights, and then gradient descent can be used for optimized search during training. At the same time, set the temperature exponential decay so that the obtained probability matrix of bit and weight selection approximates the one-hot encoding, enabling the selection of bits and weights to converge to a single selection; S4. Select the maximum probability bit width in the bit search result to construct a lightweight network, and save the weight parameters obtained with the maximum probability bit width during joint search as the weight parameters of the lightweight network. Then, conduct weight search to optimize the lightweight network model and output the compressed and optimized lightweight network model; For one layer of the convolutional neural network model, assuming there are a total of N selectable bit numbers, the selection probability of searching for the m-th bit is calculated by the following formula: where p s represents the bit selection probability, and further represents the selection probability of searching for the m-th bit, N is the number of types of selectable bits, is the selection weight of the m-th bit before normalization, τ0 is the set decay temperature, is the selection weight of the j-th bit before normalization; For one layer of the convolutional neural network model, assume there are a total of N selectable bit numbers, and assume that the m-th bit number corresponds to v m selectable quantized weight values. Then, the selection probability of the i-th quantized weight value of the m-th bit number in the search is calculated by the following formula: Among them, p represents the selection probability of the quantization weight value, and subscripts are further added. represents the selection probability of the i-th quantization weight value of the m-th bit number in the search; v m represents the number of available quantization weight values corresponding to the m-th bit number; is a learnable parameter, which is the selection weight before normalization of the j-th quantization weight value of the m-th bit number, and τ1 is a preset decay temperature. is the selection weight before normalization of the i-th quantization weight value of the m-th bit number.

2. The model compression method combining bit quantization and bit weight search according to claim 1, characterized in that The preprocessing includes padding, cropping, and image enhancement of the image data.

3. A model compression method combining bit quantization and bit weight search according to claim 1, characterized in that Cut the loaded image data into a training set, a validation set, and a test set.

4. A model compression method combining bit quantization and bit weight search according to claim 1, characterized in that Use the gradient search strategy to conduct search training on the convolutional neural network model, and alternately search for the bit limit and bit weight value of the convolutional neural network model in the training set and the validation set so that the selection of bits and weights can be the overall best.

5. A model compression method combining bit quantization and bit weight search according to claim 1, characterized in that The output function during the training of the convolutional neural network model is: Y = σ(W * X + b) (3) Among them, Y represents the output during model training, σ(·) is the activation function, X is the input of the convolutional layer, b is the bias vector of the convolutional layer, and W is the convolutional weight during training, and its elements are: Among them, represents the selection probability of searching for the m-th bit; N is the number of types of selectable bits; is the consecutive weight element under the m-th quantization bit, which corresponds to v m selectable quantization weight values, which are calculated by the following formula: Among them, is the selection probability of the i-th quantization weight value of the m-th bit number in the search, V is the set of bit weight search quantization values. For the set V of bit weight search quantization values of q bits, there are 2 q discrete weight values. Therefore, v i is the i-th normalized bit weight quantization value, is the 2 q th discrete weight value.

6. A model compression method combining bit quantization and bit weight search according to claim 1, characterized in that The output function during the inference of the convolutional neural network model is: Y = σ(W q *X + b) (7) where Y represents the output during model training, σ(·) is the activation function, X is the input of the convolutional layer, b is the bias vector of the convolutional layer, and W q is the convolutional weight during training, and its elements are: Among them, J (m) represents the value at the m-th position of the one-hot vector J, is the discrete bit weight under the m-th quantization bit, and it corresponds to v m available quantization weight values, and its calculation formula is as follows: Among them, I m represents the position corresponding to the maximum value of the selection probability of the quantization weight value of the m-th bit number in the search, that is, a vector of length v m where the element corresponding to the position of the maximum selection weight is 1 and the rest are 0; represents the value of the j-th position of the one-hot vector I m ; v j represents the j-th normalized bit weight quantization value in the set V of bit weight search quantization values; is the selection of the i-th quantization weight value of the m-th bit number in the search; J represents the one-hot vector corresponding to the position of the maximum value of the bit number selection probability in the search, and its calculation formula is as follows: Since there are a total of N available bit numbers, J is a vector of length N, where the element corresponding to the position of the maximum selection weight is 1 and the rest are 0. represents the selection probability of searching for the i-th bit.

7. A model compression method combining bit quantization and bit weight search according to any one of claims 1 to 6, characterized in that During search training, add a loss term for computing resources and a loss term for storage resources to the corresponding loss function to simultaneously optimize the computing cost and storage cost. The obtained loss function is: L = L c + β compute · L compute + γ memory · L memory (12) where L c is the cross-entropy loss of the convolutional neural network model, L compute is the number of bit operations used by the convolutional neural network model, β compute is the adjustable weight corresponding to the total computational cost, L memory is the total storage involved in the convolutional neural network model, γ memory is the adjustable weight corresponding to the total storage.

Images