A method and device for quantizing model parameters of convolutional neural network and related devices

Abstract

The present application discloses a method for quantizing model parameters of a convolutional neural network, including: obtaining a data set for a digital classification task; constructing a convolutional neural network for the digital classification task; using the data set to train the convolutional neural network, and the trained convolutional neural network is used to perform the digital classification task; wherein, during the forward propagation process of the quantized convolution layer, the model parameters are quantized based on a sign function; the model parameters include weights and activation values; during the backward propagation process of the quantized convolution layer, the model parameters are quantized based on a parameterized hyperbolic tangent function; wherein the parameters in the hyperbolic tangent function are determined based on the data distribution of the model parameters of the quantized convolution layer, and the present application can improve the model accuracy of digital classification. The present application also discloses a device for quantizing model parameters of a convolutional neural network, a computer-readable storage medium, and an electronic device, which have the above beneficial effects.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a method and device for quantizing model parameters of a convolutional neural network and related devices. Background Art

[0002] In recent years, deep learning technology has made great progress in the field of computer vision (for example, digital classification, object detection, instance segmentation, and face recognition), and has been widely used in various practical scenarios. In order to achieve high performance (high precision), engineers usually use neural networks with a depth of dozens or even hundreds of layers (for example, residual networks ResNet50 or ResNet100) as the backbone network to design various computer vision algorithms. In practical applications, application servers are equipped with high-performance GPU cards to complete model inference tasks. However, without the support of high-performance GPU cards, the application of these computer vision algorithms will be limited.

[0003] How to reduce storage and computation, that is, model lightweighting, is a widely concerned technical field. Typical methods for lightweighting deep convolutional neural network models include: model pruning, low-rank matrix decomposition, low-bit quantization, etc. Among these methods, low-bit quantization technology (especially binary quantization technology) is the most effective model lightweight technology. Binarized convolutional neural networks use bit-wise operations to replace floating-point operations, and use bit-wise operation instructions of CPU, GPU or other special devices to significantly accelerate the model reasoning speed.

[0004] Although using the above approximate function instead of the derivative of the sign function in the back propagation process can solve the gradient vanishing problem in the training process, such a design leads to a mismatch between the forward function and the reverse function. The mismatch between the forward function and the reverse function will lead to low model accuracy.

[0005] Therefore, how to improve the model accuracy of digital classification is a technical problem that technicians in this field currently need to solve. Summary of the invention

[0006] The purpose of this application is to provide a method and device for quantizing model parameters of a convolutional neural network, a computer-readable storage medium and an electronic device, which can improve the model accuracy of digital classification.

[0007] In order to solve the above technical problems, the present application provides a method for quantizing model parameters of a convolutional neural network, comprising:

[0008] Obtain a data set for a digital classification task; wherein the data set includes digital images and corresponding digital labels;

[0009] Constructing a convolutional neural network for a digital classification task; wherein the convolutional neural network includes a floating-point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer, and a loss calculation layer;

[0010] The convolutional neural network is trained using the data set, and the trained convolutional neural network is used to perform a digital classification task;

[0011] Wherein, in the forward propagation process of the quantized convolution layer, the model parameters are quantized based on the sign function; wherein the model parameters include weights and activation values;

[0012] During the back propagation process of the quantized convolutional layer, the model parameters are quantized based on a parameterized hyperbolic tangent function; wherein the parameters in the hyperbolic tangent function are determined based on the data distribution of the model parameters of the quantized convolutional layer.

[0013] Optionally, the convolutional neural network includes a plurality of sequentially connected structural units, a fully connected layer, a classification layer, and a loss calculation layer;

[0014] The structural units include a first type of structural unit, a second type of structural unit and a third type of structural unit; the first type of structural unit includes a floating-point convolution layer, a normalization layer and a nonlinear layer connected in sequence; the second type of structural unit includes a quantized convolution layer, a normalization layer and a nonlinear layer connected in sequence; the third type of structural unit includes a quantized convolution layer, a pooling layer, a normalization layer and a nonlinear layer connected in sequence.

[0015] Optionally, in the forward propagation process of the quantized convolution layer, the model parameters are quantized based on a sign function, including:

[0016] Calculating intermediate quantized values ​​corresponding to the model parameters based on a sign function, and calculating scaling factors corresponding to the model parameters;

[0017] The product of the intermediate quantization value and the scaling factor is determined as the final quantization value of the model parameter.

[0018] Optionally, after quantizing the model parameters based on the sign function, the method further includes:

[0019] An inner product operation of a bit operation is performed on the quantized value corresponding to the weight and the quantized value corresponding to the activation value to obtain the output of the quantized convolution layer.

[0020] Optionally, the quantizing the model parameters based on a parameterized hyperbolic tangent function includes:

[0021] Calculating the quantized value corresponding to the model parameter based on the target hyperbolic tangent function;

[0022] Wherein, the target hyperbolic tangent function is specifically: y(x)=c0*tanh(c1*x);

[0023] Wherein, x is the model parameter, y(x) is the quantized value, c0 and c1 are hyperparameters, and c1 is determined based on the data distribution of the model parameters of the quantized convolutional layer.

[0024] Optionally, also include:

[0025] Calculate c1 under the target constraint condition; wherein the target constraint condition is specifically that the ratio of the number of target model parameters that meet the preset condition to the total number of model parameters is greater than or equal to the preset ratio r, and the preset condition is specifically tanh(c1*x)∈(-1,-r)∪(1,r).

[0026] Optionally, also include:

[0027] Determine the data distribution corresponding to the model parameters of the quantized convolutional layer in different intervals.

[0028] The present application also provides a model parameter quantization device for a convolutional neural network, the device comprising:

[0029] An acquisition module, used to acquire a data set for a digital classification task; wherein the data set includes digital images and corresponding digital labels;

[0030] A construction module for constructing a convolutional neural network for a digital classification task; wherein the convolutional neural network includes a floating-point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer, and a loss calculation layer;

[0031] A first quantization module, used for quantizing model parameters based on a sign function during the forward propagation process of the quantized convolution layer; wherein the model parameters include weights and activation values;

[0032] A second quantization module, configured to quantize the model parameters based on a hyperbolic tangent function during the back propagation process of the quantized convolution layer; wherein the parameters in the hyperbolic tangent function are determined based on the distribution of the model parameters of the quantized convolution layer;

[0033] A training module is used to train the convolutional neural network using the data set, and the trained convolutional neural network is used to perform a digital classification task.

[0034] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the above-mentioned convolutional neural network model parameter quantization method.

[0035] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps of executing the above-mentioned convolutional neural network model parameter quantization method are implemented.

[0036] The present application provides a method for quantizing model parameters of a convolutional neural network, comprising: obtaining a data set for a digital classification task; wherein the data set contains digital images and corresponding digital labels; constructing a convolutional neural network for the digital classification task; wherein the convolutional neural network includes a floating-point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer, and a loss calculation layer; training the convolutional neural network using the data set, and the trained convolutional neural network is used to perform the digital classification task; wherein, in the forward propagation process of the quantized convolution layer, the model parameters are quantized based on a sign function; wherein the model parameters include weights and activation values; in the backward propagation process of the quantized convolution layer, the model parameters are quantized based on a parameterized hyperbolic tangent function; wherein the parameters in the hyperbolic tangent function are determined based on the data distribution of the model parameters of the quantized convolution layer.

[0037] The present application obtains a data set for a digital classification task and constructs a convolutional neural network for the digital classification task. After training the convolutional neural network, the trained convolutional neural network is used to perform the digital classification task. In the forward propagation process of the quantized convolutional layer, the present application quantizes the model parameters based on a sign function; in the back propagation process of the quantized convolutional layer, the model parameters are quantized based on a parameterized hyperbolic tangent function. The quantization function of the present application is dynamically determined based on the data distribution of the weight values ​​and activation values ​​of the training process, and different quantization functions are used for the weight values ​​and activation values ​​at each layer. Therefore, the present application can improve the model accuracy of digital classification. The present application also provides a model parameter quantization device for a convolutional neural network, a computer-readable storage medium, and an electronic device, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flowchart of a method for quantizing model parameters of a convolutional neural network provided in an embodiment of the present application;

[0040] Figure 2A schematic diagram of the structure of the first type of structural unit (F-type unit) provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of the second type of structural unit (A-type unit) provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of the third type of structural unit (type B unit) provided in the embodiment of the present application;

[0043] Figure 5 A convolutional neural network structure diagram for training a digital classification system provided in an embodiment of the present application;

[0044] Figure 6 Schematic diagram of the y(x) function when c0 and c1 take different values;

[0045] Figure 7 Schematic diagram of y'(x) function when c0 and c1 take different values;

[0046] Figure 8 A schematic diagram of the structure of a model parameter quantization device for a convolutional neural network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] See below Figure 1 , Figure 1 A flowchart of a method for quantizing model parameters of a convolutional neural network provided in an embodiment of the present application.

[0049] Specific steps may include:

[0050] S101: Obtain a data set for a digital classification task;

[0051] The data set contains digital images and corresponding digital labels. For digital classification tasks, the data set is images and category labels (annotated true values). For example: for digital classification problems from 0 to 9, the input set contains color images or grayscale images from 0 to 9, the size of each image is 32*32 pixels, and the label of each image is the corresponding number. The training samples are divided into 10 subsets, each subset corresponds to a number, and each subset contains at least one image. In order to train a high-precision model, each subset should contain enough samples (images), such as thousands of samples or tens of thousands of samples. In addition, the training samples should be diversified, for example: including background changes, font changes, lighting changes, etc.

[0052] S102: Constructing a convolutional neural network for digital classification tasks;

[0053] The convolutional neural network includes a floating-point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer and a loss calculation layer.

[0054] As a preferred implementation, the convolutional neural network constructed in this embodiment includes multiple sequentially connected structural units, fully connected layers, classification layers and loss calculation layers; the structural units include first-type structural units, second-type structural units and third-type structural units; the first-type structural units include sequentially connected floating-point convolution layers, normalization layers and non-linear layers; the second-type structural units include sequentially connected quantized convolution layers, normalization layers and non-linear layers; the third-type structural units include sequentially connected quantized convolution layers, pooling layers, normalization layers and non-linear layers.

[0055] For the first type of structural unit, see Figure 2 ,The first type of structural unit (F-type unit) contains a floating point convolution layer, a normalization layer (Batch Normalization) and a non-linear layer. Figure 2 The arrows in the middle indicate the direction of data flow. Figure 3 ,Schematic diagram of the structure of the second type of structural unit (Type A unit).,The basic structural unit (Type A unit) includes a quantized convolution layer, a normalization layer (Batch Normalization) and a nonlinear layer. Figure 3 The arrows in the middle indicate the direction of data flow. For the third type of structural unit, see Figure 4 ,Schematic diagram of the structure of the third type of structural unit (B-type unit). The basic structural unit (B-type unit) contains a quantized convolution layer, a pooling layer, a normalization layer (Batch Normalizatin) and a nonlinear layer. Figure 4 The arrows in the middle indicate the direction of data flow.

[0056] S103: Using the data set to train the convolutional neural network, the trained convolutional neural network is used to perform a digital classification task;

[0057] In the forward propagation process of the quantized convolution layer, the model parameters are quantized based on a sign function; wherein the model parameters include weights and activation values. In the backward propagation process of the quantized convolution layer, the model parameters are quantized based on a parameterized hyperbolic tangent function; wherein the parameters in the hyperbolic tangent function are determined based on the data distribution of the model parameters of the quantized convolution layer.

[0058] Specifically, the process of quantizing model parameters based on a sign function includes: calculating an intermediate quantization value corresponding to the model parameter based on a sign function, and calculating a scaling factor corresponding to the model parameter; and determining the product of the intermediate quantization value and the scaling factor as the final quantization value of the model parameter.

[0059] After the model parameters are quantized based on the sign function, an inner product operation of a bit operation may be performed on the quantized value corresponding to the weight and the quantized value corresponding to the activation value to obtain the output of the quantized convolution layer.

[0060] In a specific implementation, the forward propagation process of the quantized convolutional layer is as follows: In this embodiment, the binary quantized weight is calculated by formula (1), where W represents the weight of a layer (expressed in floating point), B w Indicates the binary quantized weight corresponding to the weight W, B w Each element in is the sign of each element in W. In order to improve the expression accuracy of the neural network, this embodiment calculates the weight scaling factor s by formula (2), ‖W‖1 represents the 1-norm of W, and n is the total number of weights. Q(W) represents the approximation of W. sign() is a sign function.

[0061] B w =sign(W); Formula (1)

[0062]

[0063] Q(W)=s*B w ; Formula (3)

[0064] This embodiment calculates the binary quantized activation value by formula (4), where A represents the activation value of a layer, and B A Indicates the binary quantized activation value corresponding to the activation value A, B A Each element in is the sign of each element in A.

[0065] Q(A)=B A =sign(A); Formula (4)

[0066] Based on the above formulas (3) and (4), this embodiment calculates the convolution operation through formula (5):

[0067] Z=Q(W)⊙Q(A)=B w ⊙B A *s; Formula (5)

[0068] Wherein, Z represents the output of the convolution, ⊙ represents the inner product operation using bit operation, and the quantization of the activation value in this embodiment can also increase the scaling factor.

[0069] Furthermore, in this embodiment, the model parameters can be quantified based on the parameterized hyperbolic tangent function in the following manner:

[0070] The quantized value corresponding to the model parameter is calculated based on the target hyperbolic tangent function; wherein the target hyperbolic tangent function is specifically: y(x)=c0*tanh(c1*x); wherein x is the model parameter, y(x) is the quantized value, c0 and c1 are hyperparameters, and c1 is determined based on the data distribution of the model parameters of the quantized convolutional layer.

[0071] In a specific implementation, the back propagation process of neural network training is as follows: This embodiment proposes to use the derivative of formula (6) (formula (9)) to approximate the derivative of the sign function sign(). When formula (6) is used for weight quantization, x represents W; when formula (6) is used for activation value quantization, x represents A.

[0072] y(x)=c0*tan h(c1*x); Formula (6)

[0073]

[0074] c0=θ(c1); Formula (5)

[0075] y'(x)=c0*c1*(1-tanh 2 (c1*x)); Formula (9)

[0076] In the above formula (6), x represents the input of the quantization function, and y(x) represents the output of the quantization function of the input x. tanh() is the hyperbolic tangent function. "*" represents element-wise multiplication. c0 and c1 are hyperparameters.

[0077] In formula (7), t represents the number of training iterations, such as from the first training iteration to the last training iteration. For example, if the total number of iterations is 500, the value range of t is an integer from 1 to 500 or an integer from 0 to 499. It means that at the time of training iteration t, c1 is determined according to the distribution of data x.

[0078] As a feasible implementation method, this embodiment can also calculate c1 under the target constraint condition; wherein the target constraint condition is specifically that the ratio of the number of target model parameters that meet the preset conditions to the total number of model parameters is greater than or equal to the preset ratio r, and the preset condition is specifically tanh(c1*x)∈(-1,-r)∪(1,r).

[0079] In a specific implementation, a ratio r is determined based on the total number of experimental iterations and the current number of iterations (for example, r = 5% at the beginning of training, r = 95% at the end of training, and r increases from 5% to 95% during training). c1 is determined based on the ratio r and the above data distribution, so that at least elements with a proportion of r are mapped to between (-1.0, -0.95) and (0.95, 1.0) through the function tanh(). 0.95 is an empirical value, and other values ​​may also be used, for example: 0.98. This embodiment recommends using a value between (0.95, 0.99).

[0080] Furthermore, this embodiment can also determine the data distribution corresponding to the model parameters of the quantized convolution layer in different intervals.

[0081] One way to calculate the above data distribution is as follows:

[0082] StepA1: Calculate the maximum value v_max of the input x of the quantization function;

[0083] StepA2: Divide the interval [0, v_max] into several bins according to the maximum value v_max;

[0084] StepA3: Project the numbers approximately zero in the input x into the interval in stepA2 to obtain a histogram of the data distribution.

[0085] Another way to calculate the above data distribution is as follows:

[0086] StepB1: Calculate the absolute value of each element in the input x of the quantization function;

[0087] StepB2: Calculate the maximum value v_max of the absolute value of each element in the input x of the quantization function;

[0088] StepB3: Divide the interval [0, v_max] into several bins according to the maximum value v_max;

[0089] StepB4: Project the absolute value of each element in the input x into the interval in stepB3 to obtain a histogram of the data distribution.

[0090] In the above formula (8), a simplified form is c0=1 / c1. Some expanded forms are c0=k / c1. k is an expansion coefficient, for example: k=2 or 3.

[0091] Formula (9) represents the derivative of formula (6) y(x). Figure 6 and Figure 7 , Figure 6 Schematic diagram of the y(x) function when c0 and c1 take different values, y(x) = c0*tanh(c1*x); Figure 7 This is a schematic diagram of the y'(x) function when c0 and c1 take different values, y'(x) = c0*c1*(1-tanh 2 (c1*x)).

[0092] This embodiment obtains a data set for a digital classification task and constructs a convolutional neural network for the digital classification task. After training the convolutional neural network, the trained convolutional neural network is used to perform the digital classification task. In the forward propagation process of the quantized convolutional layer, this embodiment quantizes the model parameters based on a sign function; in the backward propagation process of the quantized convolutional layer, the model parameters are quantized based on a parameterized hyperbolic tangent function. The quantization function of this embodiment is dynamically determined based on the data distribution of the weight values ​​and activation values ​​of the training process, and different quantization functions are used for the weight values ​​and activation values ​​at each layer, so this embodiment can improve the model accuracy of digital classification.

[0093] The process described in the above embodiment is explained below through an embodiment in actual application.

[0094] In a general convolutional neural network, the model parameters of the convolutional neural network (for example: the weight value of the convolutional layer in the convolutional neural network) are expressed by floating point numbers; the activation values ​​generated by the convolutional neural network during operation are also expressed by floating point numbers. The model parameters generated by the convolutional neural network quantization algorithm of the present invention only contain two numerical values ​​+1 and -1; the activation values ​​generated during the operation of the convolutional neural network are also expressed by +1 and -1. Under such a configuration, the convolution operation of the convolutional neural network can be implemented using a "bitwise operation", thereby accelerating the reasoning operation of the convolutional neural network and saving storage overhead during the reasoning process of the convolutional neural network. Operation refers to operations such as convolution and pooling in a neural network.

[0095] In a typical binary convolutional neural network training, during the forward propagation process, the floating-point weight values ​​and floating-point activation values ​​are quantized to binary values ​​{-1, +1} using a piecewise function. The most commonly used piecewise function is the following sign function sign(x).

[0096]

[0097] Because the derivative of the above symbolic function is Almost everywhere is zero, so the gradient in the back propagation process will disappear. The vanishing gradient will cause the neural network training process to fail to converge and fail to train a high-precision model.

[0098]

[0099] In order to solve the gradient disappearance problem, the derivative of the above symbolic function is usually approximated by the following function:

[0100]

[0101] Although using the above approximate function instead of the derivative of the sign function in the back propagation process can solve the gradient vanishing problem in the training process, such a design leads to a mismatch between the forward function and the reverse function. The mismatch between the forward function and the reverse function will lead to low model accuracy.

[0102] There is a binary convolutional neural network quantization method in the related art, which proposes a parameterized hyperbolic tangent function tanh(v*x) as a quantization function, where v is a hyperparameter and x is an independent variable. At the beginning of training, v takes a very small value (for example, v=1). During the training iteration, the v value gradually increases to a very large value (for example: v=1000), and tanh(v*x) will approach the sign function sign(x).

[0103]

[0104] The differential of the above function is:

[0105]

[0106] When the training is finished, the model parameters are quantized using the sign function to obtain a binary convolutional neural network model. In the process of model inference, the above literature uses the sign function to quantize the activation value to obtain a binary activation value, and then uses bit operations to perform convolution operations.

[0107] The above method has the following two problems: ① The v value is set based on the experience of the development engineer without considering the data distribution characteristics of the weight value and the activation value; ② The above method uses the same "quantization function" in each layer of the entire network without considering the data distribution characteristics of each layer. The above two problems will lead to unstable neural network training process and low model accuracy.

[0108] In order to solve the above technical problems, the embodiment of the present application provides a convolutional neural network quantization scheme. During the training process of the convolutional neural network, the weight values ​​and activation values ​​are quantized using the scheme proposed in this embodiment to generate a binary convolutional neural network. The present embodiment proposes an adaptive quantization function, and the parameters of the quantization function are dynamically determined based on the data distribution of the weight values ​​of the convolutional neural network model and the data distribution of the activation values ​​of the convolutional neural network, without relying on the personal experience of the engineer. Furthermore, the present embodiment proposes a layer-by-layer quantization solution, using a specific quantization function for each layer. Utilizing the above two inventive points, the present embodiment can improve the accuracy of the trained binary convolutional neural network and accelerate the convergence speed of the binary convolutional neural network. The specific implementation method of this embodiment can be a digital classification task, which can specifically include the following steps:

[0109] Step 1: Input the task-related data set.

[0110] Among them, the data set related to the above tasks includes a training set and annotated true values.

[0111] Step 2: Build a neural network.

[0112] The convolutional neural network of this embodiment is composed of a convolutional layer, a batch processing layer, a pooling layer, a fully connected layer and a nonlinear layer.

[0113] See also Figure 5 , Figure 5 A convolutional neural network structure diagram for training a digital classification system provided in an embodiment of the present application, using the above Figure 2 , Figure 3 and Figure 4 The three basic building blocks shown can be constructed Figure 5 A neural network for digit classification shown.

[0114] A1 structural unit includes:

[0115] Floating-point convolution layer: The convolution layer is a floating-point convolution operation with parameters of 3*(3*3)*128. The first 3 represents the number of channels of the input image (when the input image is a color image, the number of channels is set to 3; when the input image is a single-channel grayscale image, the number of channels is set to 1); (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3; 128 represents the total number of convolution kernels, which is also the number of channels of the output layer. In order to keep the size of the output feature map consistent with the input image, the input image is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0116] Normalization layer: The normalization layer is the Batch Normalization layer.

[0117] Nonlinear layer: This embodiment uses HardTanh as the nonlinear layer. The mathematical expression of the nonlinear layer is:

[0118]

[0119] In the above formula, x represents input.

[0120] A2 structural units include:

[0121] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 128*(3*3)*128, where the first 128 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 128 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0122] Normalization layer: The normalization layer is the Batch Normalization layer.

[0123] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0124] The B1 structural unit includes:

[0125] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 128*(3*3)*128, where the first 128 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 128 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0126] Pooling layer: The present invention uses a max-pooling operation, where the pooling kernel is 2*2 and the step size is 2. The resolution of the feature map output by the pooling layer is 1 / 2 of the resolution of the input feature map.

[0127] Normalization layer: The normalization layer is the Batch Normalization layer.

[0128] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0129] A3 structural units include:

[0130] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 128*(3*3)*256, where the first 128 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 256 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0131] Normalization layer: The normalization layer is the Batch Normalization layer.

[0132] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0133] A4 structural units include:

[0134] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 256*(3*3)*256, where the first 256 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 256 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0135] Normalization layer: The normalization layer is the Batch Normalization layer.

[0136] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0137] The B2 structural unit includes:

[0138] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 256*(3*3)*256, where the first 256 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 256 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0139] Pooling layer: This embodiment uses the maximum pooling operation, where: the pooling kernel is 2*2, and the step size is 2. The resolution of the feature map output by the pooling layer is 1 / 2 of the resolution of the input feature map.

[0140] Normalization layer: The normalization layer is the Batch Normalization layer.

[0141] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0142] A5 structural unit includes:

[0143] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 256*(3*3)*512, where the first 256 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 512 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0144] Normalization layer: The normalization layer is the Batch Normalization layer.

[0145] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0146] A6 structural unit includes:

[0147] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 512*(3*3)*512, where the first 512 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 512 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0148] Normalization layer: The normalization layer is the Batch Normalization layer.

[0149] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0150] A7 structural unit includes:

[0151] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 512*(3*3)*512, where the first 512 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 512 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0152] Normalization layer: The normalization layer is the Batch Normalization layer.

[0153] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0154] The B3 structural unit includes:

[0155] Quantized convolution layer: The convolution layer is a quantized convolution operation with parameters of 512*(3*3)*512, where the first 512 represents the number of channels of the output feature map of the previous layer; (3*3) represents the convolution kernel, the width of the convolution kernel is 3 and the height of the convolution kernel is also 3. 512 represents the total number of convolution kernels. In order to keep the size of the output feature map consistent with the input image, the input feature map is padded with zeros during the convolution operation, and the step size of the convolution operation is 1.

[0156] Pooling layer: This embodiment uses the maximum pooling operation, where: the pooling kernel is 2*2, and the step size is 2. The resolution of the feature map output by the pooling layer is 1 / 2 of the resolution of the input feature map.

[0157] Normalization layer: The normalization layer is the Batch Normalization layer.

[0158] Non-linear layer: This embodiment uses HardTanh as the non-linear layer.

[0159] The fully connected layer is a floating point fully connected layer operation with parameters of 512*4*4*10, where 512 corresponds to the number of channels of the output feature map of the B3 layer; 4*4 corresponds to the width and height of the output feature map of the B3 layer; 10 corresponds to the total number of digital categories (that is, 10 numbers). The Softmax layer corresponds to the softmax function. The loss layer: corresponds to the digital classification problem, and the cross entropy loss is used as the loss function.

[0160] Step 3: Train the neural network.

[0161] This embodiment uses SGD(m) to optimize the weight parameters. The initial learning rate is 0.01, the momentum is 0.9, the weight decay is 1*10-4, the total number of training epochs is 500, and the learning rate decreases by 1 / 10 at the 250th epoch and the 400th epoch. SGD / SGD(m) refers to stochastic gradient descent / stochastic gradient descent with momentum. This embodiment uses the training rules of conventional deep neural networks to train other operations other than the quantized convolution layer. During the forward propagation of the quantized convolution layer, the model parameters are quantized based on a sign function; wherein the model parameters include weights and activation values. During the back propagation of the quantized convolution layer, the model parameters are quantized based on a parameterized hyperbolic tangent function; wherein the parameters in the hyperbolic tangent function are determined based on the data distribution of the model parameters of the quantized convolution layer.

[0162] Step 4: Neural network inference.

[0163] When the neural network training is completed, save the binary quantized weights B of the quantized convolution layer. wThe present invention uses conventional deep neural network operations to process other layers except the quantized convolution layer. In the quantized convolution layer, when the neural network inference is performed, when the previous layer outputs the activation value A, sign() is used to convert A into B A , and then use formula (5) to perform convolution operation.

[0164] The differences between this embodiment and the related art are as follows: the quantization function of this embodiment is dynamically determined based on the data distribution of weight values ​​and activation values ​​during the training process; the related art presets values ​​based on the experience of engineers. This embodiment uses different quantization functions for weight values ​​and activation values ​​at each layer; the related art uses the same quantization function. This embodiment considers the data distribution characteristics of different layers and uses different quantization functions for each layer. The related art uses the same quantization function at different layers.

[0165] This embodiment proposes a new binary quantization scheme, which makes the training process more stable and the model more accurate by introducing a dynamic quantization mechanism and a layer-by-layer quantization mechanism. The method proposed in the present invention is not limited to specific tasks, and can be used in image classification tasks, target detection, and image segmentation tasks.

[0166] See also Figure 8 , Figure 8 A schematic diagram of the structure of a device for quantizing model parameters of a convolutional neural network provided in an embodiment of the present application;

[0167] The device may include:

[0168] The acquisition module 801 is used to acquire a data set for a digital classification task; wherein the data set includes digital images and corresponding digital labels;

[0169] A construction module 802 is used to construct a convolutional neural network for a digital classification task; wherein the convolutional neural network includes a floating point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer, and a loss calculation layer;

[0170] A first quantization module 803, used for quantizing model parameters based on a sign function during the forward propagation process of the quantized convolution layer; wherein the model parameters include weights and activation values;

[0171] A second quantization module 804 is used to quantize the model parameters based on a hyperbolic tangent function during the back propagation process of the quantized convolution layer; wherein the parameters in the hyperbolic tangent function are determined based on the distribution of the model parameters of the quantized convolution layer;

[0172] The training module 805 is used to train the convolutional neural network using the data set, and the trained convolutional neural network is used to perform a digital classification task.

[0173] This embodiment obtains a data set for a digital classification task and constructs a convolutional neural network for the digital classification task. After training the convolutional neural network, the trained convolutional neural network is used to perform the digital classification task. In the forward propagation process of the quantized convolutional layer, this embodiment quantizes the model parameters based on a sign function; in the backward propagation process of the quantized convolutional layer, the model parameters are quantized based on a parameterized hyperbolic tangent function. The quantization function of this embodiment is dynamically determined based on the data distribution of the weight values ​​and activation values ​​of the training process, and different quantization functions are used for the weight values ​​and activation values ​​at each layer, so this embodiment can improve the model accuracy.

[0174] Furthermore, the convolutional neural network includes a plurality of sequentially connected structural units, a fully connected layer, a classification layer, and a loss calculation layer;

[0175] The structural units include a first type of structural unit, a second type of structural unit and a third type of structural unit; the first type of structural unit includes a floating-point convolution layer, a normalization layer and a nonlinear layer connected in sequence; the second type of structural unit includes a quantized convolution layer, a normalization layer and a nonlinear layer connected in sequence; the third type of structural unit includes a quantized convolution layer, a pooling layer, a normalization layer and a nonlinear layer connected in sequence.

[0176] Furthermore, the first quantization module 803 is used to calculate the intermediate quantization value corresponding to the model parameter based on the sign function, and calculate the scaling factor corresponding to the model parameter; and is also used to determine the product of the intermediate quantization value and the scaling factor as the final quantization value of the model parameter.

[0177] Furthermore, it also includes:

[0178] The quantized convolution layer output module is used to perform an inner product operation of a bit operation on the quantized value corresponding to the weight and the quantized value corresponding to the activation value after the model parameters are quantized based on the sign function, so as to obtain the output of the quantized convolution layer.

[0179] Furthermore, the second quantization module 804 is used to calculate the quantization value corresponding to the model parameter based on the target hyperbolic tangent function; wherein, the target hyperbolic tangent function is specifically: y(x)=c0*tanh(c1*x); wherein, x is the model parameter, y(x) is the quantization value, c0 and c1 are hyperparameters, and c1 is determined based on the data distribution of the model parameters of the quantized convolutional layer.

[0180] Furthermore, it also includes:

[0181] A calculation module is used to calculate c1 under a target constraint condition; wherein the target constraint condition is specifically that the ratio of the number of target model parameters that meet the preset condition to the total number of model parameters is greater than or equal to a preset ratio r, and the preset condition is specifically tanh(c1*x)∈(-1,-r)∪(1,r).

[0182] Furthermore, it also includes:

[0183] The data distribution determination module is used to determine the data distribution corresponding to the model parameters of the quantized convolutional layer in different intervals.

[0184] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0185] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0186] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.

[0187] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0188] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for quantizing model parameters of a convolutional neural network, characterized in that: include: Obtain a data set for a digital classification task; wherein the data set includes digital images and corresponding digital labels; Constructing a convolutional neural network for a digital classification task; wherein the convolutional neural network includes a floating-point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer, and a loss calculation layer; The convolutional neural network is trained using the data set, and the trained convolutional neural network is used to perform a digital classification task; wherein, in the forward propagation process of the quantized convolutional layer, the model parameters are quantized based on a sign function; wherein the model parameters include weights and activation values; In the back propagation process of the quantized convolution layer, the model parameters are quantized based on a parameterized hyperbolic tangent function; wherein the parameters in the hyperbolic tangent function are determined based on the data distribution of the model parameters of the quantized convolution layer; Wherein, the quantizing of the model parameters based on the parameterized hyperbolic tangent function includes: Calculating the quantized value corresponding to the model parameter based on the target hyperbolic tangent function; Wherein, the target hyperbolic tangent function is specifically: y(x)=c0*tanh(c1*x); Among them, x is the model parameter, y(x) is the quantized value, c0 and c1 are hyperparameters, c1 is calculated under the target constraint condition, and the target constraint condition is specifically that the ratio of the number of target model parameters that meet the preset condition to the total number of model parameters is greater than or equal to the preset ratio r, and the preset condition is specifically tanh(c1*x)∈(-1,-r)∪(r,1).

2. The model parameter quantization method according to claim 1, characterized in that: The convolutional neural network includes a plurality of sequentially connected structural units, a fully connected layer, a classification layer and a loss calculation layer; The structural unit includes a first type structural unit, a second type structural unit and a third type structural unit; the first type structural unit includes a sequentially connected floating point convolution layer, a normalization layer and a nonlinear layer; the second type structural unit includes a sequentially connected quantized convolution layer, a normalization layer and a nonlinear layer; The third type structural unit includes a quantized convolution layer, a pooling layer, a normalization layer and a nonlinear layer connected in sequence.

3. The model parameter quantization method according to claim 1, characterized in that: In the forward propagation process of the quantized convolutional layer, the model parameters are quantized based on the sign function, including: Calculating intermediate quantized values ​​corresponding to the model parameters based on a sign function, and calculating scaling factors corresponding to the model parameters; The product of the intermediate quantization value and the scaling factor is determined as the final quantization value of the model parameter.

4. The model parameter quantization method according to claim 1, characterized in that: After the model parameters are quantized based on the symbolic function, the method further includes: An inner product operation of a bit operation is performed on the quantized value corresponding to the weight and the quantized value corresponding to the activation value to obtain the output of the quantized convolution layer.

5. The model parameter quantization method according to claim 1, characterized in that: Also includes: Determine the data distribution corresponding to the model parameters of the quantized convolutional layer in different intervals.

6. A model parameter quantization device for a convolutional neural network, characterized in that: include: An acquisition module, used to acquire a data set for a digital classification task; wherein the data set includes digital images and corresponding digital labels; A construction module for constructing a convolutional neural network for a digital classification task; wherein the convolutional neural network includes a floating-point convolution layer, a quantized convolution layer, a pooling layer, a normalization layer, a nonlinear layer, a fully connected layer, a classification layer, and a loss calculation layer; A first quantization module, used for quantizing model parameters based on a sign function during the forward propagation process of the quantized convolution layer; wherein the model parameters include weights and activation values; A second quantization module, configured to quantize the model parameters based on a hyperbolic tangent function during the back propagation process of the quantized convolution layer; wherein the parameters in the hyperbolic tangent function are determined based on the distribution of the model parameters of the quantized convolution layer; A training module, used to train the convolutional neural network using the data set, and the trained convolutional neural network is used to perform a digital classification task; The second quantization module is specifically used to: calculate the quantization value corresponding to the model parameter based on the target hyperbolic tangent function; Wherein, the target hyperbolic tangent function is specifically: y(x)=c0*tan(c1*x); Wherein, x is the model parameter, y(x) is the quantized value, c0 and c1 are hyperparameters, c1 is calculated under the target constraint condition, and the target constraint condition is specifically that the ratio of the number of target model parameters that meet the preset condition to the total number of model parameters is greater than or equal to the preset ratio r, and the preset condition is specifically tanh(c1*x)∈(-1,-r)∪(r,1).

7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for quantizing model parameters of a convolutional neural network as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for quantizing model parameters of a convolutional neural network as described in any one of claims 1 to 5 are implemented.

Images