Convolutional neural network quantitative training method, apparatus and device, and medium

Abstract

The invention provides a convolutional neural network quantitative training method, which can be applied to the technical field of neural network training, and comprises the following steps: obtaining an initial convolutional neural network; based on a quantitative perception training method, training the convolutional layer and the full connection layer to obtain quantized weights; judging whether the weight meets an optimization target or not according to a preset condition; inputting the weights which do not meet the optimization target into a penalty function to obtain penalty loss; adding the penalty loss and the task loss according to a preset weight to obtain a total loss function; performing joint training based on the total loss function; and fusing the parameters of the batch normalization layer into a preceding-stage convolutional layer to obtain a trained convolutional neural network. On the basis of quantized perception training, quantized weight distribution is constrained, and the absolute value of the sum of the multiply-accumulate parts of a convolutional layer and a full-connection layer is reduced, so that the data overflow condition on an accumulator with a fixed bit width is reduced, and the target precision of a quantization algorithm on a deployment platform is improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network training technology, and in particular to a convolutional neural network quantization training method, device, equipment and medium. Background Art

[0002] With the development of convolutional neural network technology and hardware based on neural network acceleration, a large number of applications deploy neural networks on dedicated chips and field programmable gate arrays (FPGAs). In order to reduce energy consumption and reduce the use of computing units, convolutional neural network algorithms deployed on this type of platform usually use quantization technology to quantize to fixed-point numbers, and use fixed-point numbers for inference on hardware. In the specific implementation process of the network convolution layer and the fully connected layer, a large number of multiplication and accumulation are required to implement convolution, and the results of the multiplication and accumulation, that is, the partial sum, need to be saved in the hardware register, that is, the accumulator. However, the bit width of the accumulator is limited. A large number of multiplication and accumulation results may cause the stored data to exceed the upper limit of the bit width representation of the accumulator, causing data overflow and thus causing errors in the results, and the functions required by the algorithm cannot be realized. Therefore, based on the needs of the hardware deployment platform, it has become an important research direction to improve the accuracy of neural networks to perceive the bit width including the accumulator during the training process of the neural network algorithm to reduce data overflow. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] In order to solve at least one of the above-mentioned technical problems of convolutional neural networks in the prior art, the embodiments of the present invention provide a convolutional neural network quantization training method, device, equipment and medium. On the basis of quantization aware training (QAT), constraints are further implemented on the quantized weight distribution, which is beneficial to reduce the absolute value of the sum of the multiplication and accumulation parts of the convolutional layer and the fully connected layer, so as to reduce the data overflow on the accumulator with a fixed bit width, and improve the target accuracy of the quantization algorithm on the deployment platform.

[0005] (II) Technical solution

[0006] In response to the above technical problems, embodiments of the present invention provide a convolutional neural network quantization training method, device, equipment and medium.

[0007] According to a first aspect of the present invention, a convolutional neural network quantization training method is provided, comprising: obtaining an initial convolutional neural network, the initial convolutional neural network comprising a convolutional layer and a fully connected layer; based on a quantization-aware training method, training the convolutional layer and the fully connected layer to obtain the quantized weights of the convolutional layer and the fully connected layer; judging whether the weights meet a preset accumulator-aware optimization target according to preset conditions; inputting the weights that do not meet the optimization target into a penalty function to obtain a penalty loss; defining the penalty loss corresponding to the weights that meet the optimization target as 0; adding the penalty loss and the task loss according to preset weights to obtain a total loss function; performing joint training based on the total loss function; and after completing all rounds of training, fusing the parameters of the batch normalization layer into the previous convolutional layer to update the scaling factor of the weight and the scaling factor of the bias in the previous convolutional layer to obtain a trained convolutional neural network.

[0008] In some exemplary embodiments, the initial convolutional neural network includes one of an initialized floating point number representation network or a pre-trained floating point number representation network.

[0009] In some exemplary embodiments, based on the quantization-aware training method, training the convolution layer and the fully connected layer includes: adding quantization nodes and dequantization nodes to the input feature maps, weights and biases of the convolution layer and the fully connected layer; based on the scaling factor of the input feature map, the scaling factor of the weight, the scaling factor of the bias and the quantization node, quantizing the input feature map, weights and biases represented by floating-point numbers using a quantization function, and converting the floating-point numbers into fixed-point integers; based on the scaling factor of the input feature map, the scaling factor of the weight and the scaling factor of the bias, converting the fixed-point integers into pseudo-quantized floating-point numbers using a dequantization node; quantizing the pseudo-quantized floating-point numbers according to the scaling factor obtained by training to obtain fixed-point quantized weights; and grouping the fixed-point quantized weights to obtain the quantized weights of the convolution layer and the fully connected layer, wherein the scaling factor of the input feature map, the scaling factor of the weight and the scaling factor of the bias participate in the training by adding a back-propagation chain, and are a learnable part of the quantization-aware training.

[0010] In some exemplary embodiments, the scaling factor of the weight is set channel by channel, and the weight of each output channel shares a scaling factor of the channel; the scaling factor of the input feature map corresponds to the entire input feature map, and the entire input feature map shares a scaling factor; and the scaling factor of the bias is calculated by the scaling factor of the weight and the scaling factor of the input feature map.

[0011] In some exemplary embodiments, the quantization function can round and truncate the input data, wherein the rounding method is determined according to the platform on which the neural network needs to be deployed; and the upper and lower bounds of the truncation are determined according to the bit width of the input data.

[0012] In some exemplary embodiments, adding the penalty loss and the task loss according to preset weights to obtain a total loss function includes: adding the penalty losses of all channels of each convolutional layer to obtain the penalty loss of one convolutional layer; adding the penalty losses of all convolutional layers to obtain a total penalty loss; and adding the total penalty loss to the task loss of the neural network according to a preset ratio to obtain a total loss function.

[0013] In some exemplary embodiments, the parameters of the batch normalization layer are integrated into the previous convolution layer to update the scaling factor of the weight and the scaling factor of the bias in the previous convolution layer, and the trained convolution neural network is obtained, including: judging the convolution neural network layer by layer and selecting the batch normalization layer; obtaining the trainable parameters of the batch normalization layer and the mean and variance of the batch normalization layer statistics; searching forward from the current batch normalization layer to find the convolution layer of the nearest neighbor of the current batch normalization layer; updating the scaling factor of the weight of the nearest neighbor convolution layer based on the trainable parameters, mean and variance to obtain the updated weight scaling factor; obtaining the pseudo-quantized floating-point weight and pseudo-quantized bias of the nearest neighbor convolution layer; and The fusion weight is calculated based on the pseudo-quantized floating-point weight, trainable parameter, mean and variance; based on the updated weight scaling factor, the fusion weight is quantized using the quantization function to obtain the updated quantized weight; based on the pseudo-quantized bias, trainable parameter, mean and variance, the fusion bias is calculated; based on the updated weight scaling factor and the scaling factor of the input feature map, the updated bias scaling factor is calculated; based on the updated bias scaling factor, the fusion bias is quantized using the quantization function to obtain the updated quantized bias; and based on the updated quantized weight and the updated quantized bias, the parameters in the convolutional neural network are updated to obtain the trained convolutional neural network.

[0014] According to a second aspect of the present invention, a convolutional neural network quantization training device is provided, comprising: an acquisition module for acquiring an initial convolutional neural network, the convolutional neural network comprising a convolutional layer and a fully connected layer; a first training module for training the convolutional layer and the fully connected layer based on a quantization-aware training method to obtain the quantized weights of the convolutional layer and the fully connected layer; a judgment module for judging whether the weights meet the preset accumulator-aware optimization target according to preset conditions; a first calculation module for inputting the weights that do not meet the optimization target into a penalty function to obtain a penalty loss; a loss definition module for defining the penalty loss corresponding to the weights that meet the optimization target as 0; a second calculation module for adding the penalty loss and the task loss according to preset weights to obtain a total loss function; a second training module for performing joint training based on the total loss function; and a third training module for fusing the parameters of the batch normalization layer into the previous convolutional layer after completing all rounds of training to update the scaling factor of the weights and the scaling factor of the bias in the previous convolutional layer to obtain a trained convolutional neural network.

[0015] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is executed by the processor to implement the above method.

[0016] According to a fourth aspect of the present invention, a storage medium is provided, characterized in that: computer-readable instructions are stored in the storage medium, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the above method.

[0017] (III) Beneficial effects

[0018] It can be seen from the above technical solutions that the convolutional neural network quantization training method, device, equipment and medium provided by the embodiments of the present invention have at least the following beneficial effects:

[0019] (1) Based on Quantization Aware Training (QAT), constraints are further imposed on the distribution of quantized weights, which is beneficial to reducing the absolute value of the sum of the multiplication and accumulation parts of the convolutional layer and the fully connected layer, thereby reducing data overflow on the fixed-bit-width accumulator and improving the target accuracy of the quantization algorithm on the deployment platform.

[0020] (2) After completing all rounds of network training, the batch normalization layer is fused. During fusion, all parameters of the batch normalization layer are integrated into the weight scaling factor of the convolutional layer. During fusion, the extreme values ​​of the quantized data will not be changed to avoid overflow. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0022] Figure 1 A schematic diagram of a process flow of a convolutional neural network quantization training method according to an embodiment of the present invention is schematically shown;

[0023] Figure 2 A schematic diagram of a process of training a convolutional layer and a fully connected layer according to an embodiment of the present invention is shown;

[0024] Figure 3 A schematic diagram of a convolutional neural network structure according to an embodiment of the present invention is shown;

[0025] Figure 4 A schematic diagram of a process of adding penalty loss and task loss according to preset weights to obtain a total loss function according to an embodiment of the present invention is shown;

[0026] Figure 5 A schematic diagram of a process of fusing the parameters of a batch normalization layer into a previous convolutional layer to obtain a trained convolutional neural network according to an embodiment of the present invention is shown;

[0027] Figure 6 A block diagram schematically shows a structure of a convolutional neural network quantization training device according to an embodiment of the present invention; and

[0028] Figure 7 A block diagram of an electronic device for a convolutional neural network quantization training method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Figure 1 A schematic diagram of a process flow of a convolutional neural network quantization training method according to an embodiment of the present invention is schematically shown;

[0031] Depend on Figure 1 It can be seen that a convolutional neural network quantization training method according to an embodiment of the present invention includes steps S110-S180.

[0032] In step S110, an initial convolutional neural network is obtained, where the initial convolutional neural network includes a convolutional layer and a fully connected layer.

[0033] In some exemplary embodiments, the initial convolutional neural network includes one of an initialized floating point number representation network or a pre-trained floating point number representation network.

[0034] For example, the initial convolutional neural network model of the training task can be a convolutional neural network structure prepared for target recognition, target detection, target tracking, image denoising, etc., depending on the task objectives. The model can be an untrained network that uses a specific method to initialize the model parameters, or it can be a pre-trained model that has been trained to a certain extent. The parameters in the model are all floating-point numbers, and they are trained into quantized models through subsequent steps.

[0035] In step S120, the convolutional layer and the fully connected layer are trained based on the quantization-aware training method to obtain quantized weights of the convolutional layer and the fully connected layer.

[0036] In some exemplary embodiments, step S120 includes steps S121-S125, see Figure 2 .

[0037] In step S121, quantization nodes and dequantization nodes are added to the input feature maps, weights, and biases of the convolutional layer and the fully connected layer.

[0038] In step S122, based on the scaling factor of the input feature map, the scaling factor of the weight, the scaling factor of the bias and the quantization node, the input feature map, weight and bias represented by floating point numbers are quantized using a quantization function, and the floating point numbers are converted into fixed-point integers, wherein the scaling factor of the input feature map, the scaling factor of the weight and the scaling factor of the bias participate in the training by adding the back-propagation chain, and are a learnable part of the quantization-aware training.

[0039] In some exemplary embodiments, the quantization function can round and truncate the input data, wherein the rounding method is determined according to the platform on which the neural network needs to be deployed; and the upper and lower bounds of the truncation are determined according to the bit width of the input data.

[0040] For example, during the training process, the input floating-point data is counted and quantized using the inserted quantization node. The floating-point number x is converted to a fixed-point integer x using the quantization node. q , to realize the quantization process, and at the same time, the scaling factor s needs to be used to control the quantization mapping, and the mapping method is:

[0041] . Functions used to implement the quantization process , which plays the role of rounding and truncating the input data. The rounding method is determined according to the platform where the neural network needs to be deployed. The upper and lower bounds that can be represented by the quantized data bit width are used as the upper and lower bounds of the truncation.

[0042] In step S123, based on the scaling factor of the input feature map, the scaling factor of the weight, and the scaling factor of the bias, a dequantization node is used to convert the fixed-point integer into a pseudo-quantized floating-point number.

[0043] For example, using the Dequantization node to quantize a fixed-point integer x q Convert to pseudo-quantized floating point number , the mapping method is: , in the actual training, during the multiplication and accumulation operations, pseudo-quantized floating-point numbers are still used to represent quantized fixed-point numbers.

[0044] In step S124, the pseudo-quantized floating-point number is quantized according to the scaling factor obtained through training to obtain a fixed-point quantization weight.

[0045] In some exemplary embodiments, the scaling factor of the weight is set channel by channel, and the weight of each output channel shares a scaling factor of the channel; the scaling factor of the input feature map corresponds to the entire input feature map, and the entire input feature map shares a scaling factor; and the scaling factor of the bias is calculated by the scaling factor of the weight and the scaling factor of the input feature map.

[0046] For example, the data that need to be quantized and pseudo-quantized include input feature maps, weights, and biases. The configurations for quantizing these three types of data are different. The scaling factors for weights are set channel by channel, that is, the weights of each output channel share a scaling factor for the channel. Therefore, for weights with m output channels, there is a set of m scaling factors. For the input feature map, only one scaling factor is used for the entire input feature map. Scaling factor used to control quantization behavior. For the bias, when quantizing the bias, the scaling factor used is jointly determined by the scaling factors of the weight and the input feature map. The scaling factors of the weight and the input feature map are set to , the scaling factor of the bias is set to , The quantization scaling factor set in the embodiment of the present invention is a learnable part in quantization-aware training, and needs to be added to the back-propagation chain to participate in the training. The straight-through estimator STE is used to perform back-propagation on the non-differentiable function.

[0047] In step S125, the fixed-point quantized weights are grouped to obtain the quantized weights of the convolutional layer and the fully connected layer.

[0048] For example, the floating-point pseudo-quantized weights are quantized according to the scaling factor obtained by training to obtain the fixed-point quantized weights (corresponding to Figure 3 Then, group the output channels according to their different values ​​to obtain the weights of the m output channels (corresponding to Figure 3 202 in the figure).

[0049] In step S130, it is determined whether the weight meets the preset accumulator-aware optimization target according to preset conditions.

[0050] For example, using a discriminator for weights of different output channels (see Figure 3 S203 in the above example performs conditional judgment. The following formula is used to judge whether the weight of the changed channel reaches the optimization target perceived by the accumulator:

[0051] (1)

[0052] Where w represents the weight of each output channel after quantization, n represents the number of weights in the output channel, N represents the bit width of the accumulator, and P represents the bit width of the quantized input feature map. Formula (1) describes the discriminator in detail (see Figure 3 The quantized weights that conform to formula (1) can reduce the multiplication and accumulation and overflow of the channel.

[0053] In step S140, the weights that do not meet the optimization target are input into the penalty function to obtain the penalty loss.

[0054] For example, add the weight of this channel to Figure 3 The penalty function at S205 in the It can be expressed as:

[0055] (2)

[0056] in, is a variable parameter that can be fine-tuned according to the experimental results. In practice, it can be empirically set to 5.5. l is the number of the current convolutional layer. Represents the weight The maximum value in Indicates the number of all weights in the current network that do not meet the judgment criteria.

[0057] In step S150, the penalty loss corresponding to the weight that meets the optimization target is defined as 0.

[0058] For example, in Figure 3 In step S204, 0 is directly added to the result position of the penalty function.

[0059] In step S160, the penalty loss and the task loss are added according to the preset weights to obtain a total loss function.

[0060] In some exemplary embodiments, step S160 includes steps S161-S163, see Figure 4 .

[0061] In step S161, the penalty losses of all channels of each convolutional layer are added together to obtain the penalty loss of one convolutional layer.

[0062] In step S162, the penalty losses of all convolutional layers are added together to obtain the total penalty loss.

[0063] In step S163, the total penalty loss is added to the task loss of the neural network according to a preset ratio to obtain a total loss function.

[0064] For example, the results of all channel penalty functions of a certain convolutional layer are Figure 3The sum is added at S206 in , and the total penalty function loss of the layer is obtained. The results of each layer are Figure 3 The total penalty loss is obtained by adding the penalty loss in S207. Figure 3 At S208 in the figure, the task loss required by the original network task is added according to a certain ratio (the task loss is Figure 3 ), that is, the total loss of the training method (see Figure 3 The total loss is calculated as:

[0065] (3)

[0066] Among them, L is the total loss of the training task, is the loss of the original task target of the network, In order to penalize the proportion of the loss in the total loss, it is fine-tuned according to the training effect and can be empirically set to 0.001 in implementation.

[0067] In step S170, joint training is performed based on the total loss function.

[0068] After step S170 is completed, if the network has not completed all rounds of training, then return to step S120; if all rounds of training have been completed, then proceed to step S180.

[0069] In step S180, after all rounds of training are completed, the parameters of the batch normalization layer are fused into the previous convolutional layer to update the scaling factors of the weights and the biases in the previous convolutional layer to obtain a trained convolutional neural network.

[0070] In some exemplary embodiments, step S180 includes steps S1801-S1811, see Figure 5 .

[0071] In step S1801, the convolutional neural network is judged layer by layer, and the batch normalization (Batch Norm, BN) layer is selected.

[0072] In step S1802, trainable parameters of the batch normalization layer and the mean and variance of the batch normalization layer statistics are obtained.

[0073] For example, record the trainable parameters of the BN layer as , and the mean of the BN layer statistics and variance .

[0074] In step S1803, a forward search is performed from the current batch normalization layer to find the convolutional layer that is the nearest neighbor of the current batch normalization layer.

[0075] In step S1804, the scaling factor of the weight of the nearest neighbor convolutional layer is updated based on the trainable parameters, the mean and the variance to obtain an updated weight scaling factor.

[0076] For example, the weight scaling factor of the convolutional layer is modified. Specifically, the weight scaling factor is Modified to :

[0077] (4)

[0078] in, is a small value to prevent the divisor from being 0. The scaling factor is modified by formula (4), where the weight is quantized per output channel, so the number of scaling factors of the weight is equal to the number of output channels, and the parameters in the BN layer are also per channel, so the scaling factor of each channel calculated here corresponds to each channel of the BN layer parameter. The fusion of weights and biases is also the same, so it will not be repeated later.

[0079] In step S1805, the pseudo-quantized floating-point weight and pseudo-quantized bias of the nearest neighbor convolution layer are obtained.

[0080] In step S1806, the fusion weight is calculated based on the pseudo-quantized floating point weight, the trainable parameter, the mean and the variance.

[0081] For example, in the process of fusion of convolutional layer weights and BN layer parameters, the floating-point weights of the convolutional layer after pseudo-quantization are extracted. , and the parameters of the BN layer are used to obtain the new fusion weights according to the following formula :

[0082] (5)

[0083] In step S1807, based on the updated weight scaling factor, the fusion weight is quantized using a quantization function to obtain an updated quantized weight.

[0084] For example, using the new fusion weight obtained in step S1806 and the new scaling factor obtained in step S1804 Requantize to get updated quantization weights :

[0085] (6)

[0086] In step S1808, a fusion bias is calculated based on the pseudo-quantized bias, trainable parameters, mean and variance.

[0087] For example, the pseudo-quantized bias After fusion with BN layer parameters, a new fusion bias is obtained :

[0088] (7)

[0089] In step S1809, an updated bias scaling factor is calculated based on the updated weight scaling factor and the scaling factor of the input feature map.

[0090] For example, after completing the offset fusion, the offset scaling factor is modified, and the offset scaling factor is changed from Modified to .

[0091] In step S1810, based on the updated offset scaling factor, the fused offset is quantized using a quantization function to obtain an updated quantized offset.

[0092] For example, the step of requantizing the bias according to the newly obtained scaling factor is that the new quantized bias is :

[0093] (8)

[0094] In step S1811, based on the updated quantization weights and the updated quantization biases, the parameters in the convolutional neural network are updated to obtain a trained convolutional neural network.

[0095] Figure 6 The structural block diagram of a convolutional neural network quantization training device according to an embodiment of the present invention is schematically shown.

[0096] like Figure 6 As shown, a convolutional neural network quantization training device 800 according to an embodiment of the present invention includes an acquisition module 810, a first training module 820, a judgment module 830, a first calculation module 840, a loss definition module 850, a second calculation module 860, a second training module 870 and a third training module 880.

[0097] The acquisition module 810 is used to acquire an initial convolutional neural network, where the convolutional neural network includes a convolutional layer and a fully connected layer.

[0098] The first training module 820 is used to train the convolutional layer and the fully connected layer based on the quantization-aware training method to obtain the quantized weights of the convolutional layer and the fully connected layer.

[0099] The judgment module 830 is used to judge whether the weight meets the preset accumulator-aware optimization target according to the preset conditions.

[0100] The first calculation module 840 is used to input the weights that do not meet the optimization target into the penalty function to obtain the penalty loss.

[0101] The loss definition module 850 defines the penalty loss corresponding to the weight that meets the optimization target as 0.

[0102] The second calculation module 860 is used to add the penalty loss and the task loss according to the preset weight to obtain the total loss function.

[0103] The second training module 870 is used to perform joint training based on the total loss function.

[0104] The third training module 880 is used to fuse the parameters of the batch normalization layer into the previous convolutional layer after completing all rounds of training, so as to update the scaling factors of the weights and the biases in the previous convolutional layer to obtain a trained convolutional neural network.

[0105] In some specific embodiments, any multiple modules among the acquisition module 810, the first training module 820, the judgment module 830, the first calculation module 840, the loss definition module 850, the second calculation module 860, the second training module 870 and the third training module 880 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0106] In some specific embodiments, at least one of the acquisition module 810, the first training module 820, the judgment module 830, the first calculation module 840, the loss definition module 850, the second calculation module 860, the second training module 870 and the third training module 880 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 810, the first training module 820, the judgment module 830, the first calculation module 840, the loss definition module 850, the second calculation module 860, the second training module 870 and the third training module 880 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0107] Figure 7 A block diagram of an electronic device for a convolutional neural network quantization training method according to an embodiment of the present invention is schematically shown.

[0108] like Figure 7 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0109] In RAM603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM602 and RAM603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM602 and / or RAM603. It should be noted that the program can also be stored in one or more memories other than ROM602 and RAM603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0110] In some specific embodiments, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0111] An embodiment of the present disclosure shows a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0112] In some specific embodiments, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0113] An embodiment of the present disclosure shows a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0114] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0115] In some specific embodiments, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0116] In some specific embodiments, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0117] In some specific embodiments, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0118] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A convolutional neural network quantization training method, characterized in that: include: Obtaining an initial convolutional neural network, wherein the initial convolutional neural network includes a convolutional layer and a fully connected layer; Based on a quantization-aware training method, the convolutional layer and the fully connected layer are trained to obtain quantized weights of the convolutional layer and the fully connected layer; Determining whether the weight meets a preset accumulator-perceived optimization goal according to preset conditions; The weights that do not meet the optimization objective are input into a penalty function to obtain a penalty loss; The penalty loss corresponding to the weight that satisfies the optimization objective is defined as 0; Add the penalty loss and task loss according to the preset weights to get the total loss function; Based on the total loss function, performing joint training; as well as After completing all rounds of training, the parameters of the batch normalization layer are fused into the previous convolutional layer to update the scaling factors of the weights and biases in the previous convolutional layer to obtain a trained convolutional neural network.

2. The method according to claim 1, characterized in that The initial convolutional neural network includes one of an initialized floating-point number represented network and a pre-trained floating-point number represented network.

3. The method according to claim 1, characterized in that The training of the convolutional layer and the fully connected layer based on the quantization-aware training method includes: Add quantization nodes and dequantization nodes to the input feature maps, weights, and biases of the convolutional and fully connected layers; Based on the scaling factor of the input feature map, the scaling factor of the weight, the scaling factor of the bias and the quantization node, the input feature map, the weight and the bias represented by floating point numbers are quantized using a quantization function to convert the floating point numbers into fixed point integers; Based on a scaling factor of the input feature map, a scaling factor of the weight, and a scaling factor of the bias, converting the fixed-point integer into a pseudo-quantized floating-point number using the dequantization node; quantizing the pseudo-quantized floating-point number according to the scaling factor obtained through training to obtain a fixed-point quantization weight; and The fixed-point quantized weights are grouped to obtain the quantized weights of the convolutional layer and the fully connected layer, Among them, the scaling factor of the input feature map, the scaling factor of the weight and the scaling factor of the bias participate in the training by joining the back-propagation chain, and are a learnable part of the quantization-aware training.

4. The method according to claim 3, characterized in that The scaling factor of the weight is set channel by channel, and the weight of each output channel shares a scaling factor of the channel; The scaling factor of the input feature map corresponds to the entire input feature map, and the entire input feature map shares one scaling factor; as well as The scaling factor of the bias is calculated by the scaling factor of the weight and the scaling factor of the input feature map.

5. The method according to claim 3, characterized in that: The quantization function can round and truncate the input data. The rounding method is determined according to the platform on which the neural network needs to be deployed; and The upper and lower bounds of the truncation are determined according to the bit width of the input data.

6. The method according to claim 1, characterized in that The total loss function obtained by adding the penalty loss and the task loss according to the preset weights includes: Add the penalty losses of all channels of each convolutional layer to get the penalty loss of one convolutional layer; Add the penalty losses of all convolutional layers to get the total penalty loss; and The total penalty loss is added to the task loss of the neural network according to a preset ratio to obtain a total loss function.

7. The method according to claim 3, characterized in that The batch normalization layer parameters are integrated into the previous convolution layer to update the weight scaling factor and the bias scaling factor in the previous convolution layer to obtain the trained convolutional neural network, including: Judge the convolutional neural network layer by layer and select the batch normalization layer; Get the trainable parameters of the batch normalization layer and the mean and variance of the batch normalization layer statistics; Search forward from the current batch normalization layer to find the convolutional layer that is the nearest neighbor of the current batch normalization layer; Update the weight scaling factor of the convolutional layer of the nearest neighbor based on the trainable parameter, the mean, and the variance to obtain an updated weight scaling factor; Obtain pseudo-quantized floating-point weights and pseudo-quantized biases of the convolution layer of the nearest neighbor; A fusion weight is calculated based on the pseudo-quantized floating-point weight, the trainable parameter, the mean, and the variance; Based on the updated weight scaling factor, quantizing the fusion weight using a quantization function to obtain an updated quantized weight; Calculating a fusion bias based on the pseudo-quantized bias, the trainable parameter, the mean, and the variance; Calculating an updated bias scaling factor based on the updated weight scaling factor and the scaling factor of the input feature map; Based on the updated bias scaling factor, quantizing the fused bias using a quantization function to obtain an updated quantized bias; and Based on the updated quantization weights and the updated quantization biases, the parameters in the convolutional neural network are updated to obtain a trained convolutional neural network.

8. A convolutional neural network quantization training device, characterized in that: The device comprises: An acquisition module, used to acquire an initial convolutional neural network, wherein the convolutional neural network includes a convolutional layer and a fully connected layer; A first training module is used to train the convolution layer and the fully connected layer based on a quantization-aware training method to obtain quantized weights of the convolution layer and the fully connected layer; A judgment module, used to judge whether the weight meets the preset accumulator-perceived optimization target according to preset conditions; A first calculation module, used for inputting the weights that do not meet the optimization target into a penalty function to obtain a penalty loss; A loss definition module defines a penalty loss corresponding to the weight that satisfies the optimization objective as 0; The second calculation module is used to add the penalty loss and the task loss according to the preset weights to obtain the total loss function; A second training module, configured to perform joint training based on the total loss function; and The third training module is used to fuse the parameters of the batch normalization layer into the previous convolutional layer after completing all rounds of training, so as to update the scaling factors of the weights and the biases in the previous convolutional layer to obtain a trained convolutional neural network.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the method according to any one of claims 1 to 7.

Images