Neural network adaptive quantization method, device, equipment, medium and product

Abstract

The present invention provides a neural network adaptive quantization method, device, equipment, medium and product, wherein the method comprises: adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusting the activation function output value based on the contribution of the activation function output value of the neural network to the output result of the neural network; performing equivalent transformation on the adjusted neural network based on the channel where the maximum value of the weight value in the adjusted neural network is located, so that the distribution of each channel of the weight value is close; performing equivalent transformation on the adjusted neural network based on the channel where the maximum value of the activation function output value in the adjusted neural network is located, so that the distribution of each channel of the activation function output value is close; performing quantization processing on the weight value and the activation function output value in the equivalently transformed neural network to obtain the target neural network. The present invention can reduce the precision loss caused by quantization without increasing the cost.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a neural network adaptive quantization method, a neural network adaptive quantization device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence, artificial neural networks (ANN), hereinafter referred to as neural networks, have achieved good results in image classification, target detection, natural language processing, etc. However, with the improvement of neural network prediction accuracy, the scale of neural networks has also increased. The increase in the scale of neural networks will occupy more storage, computing, bandwidth and other resources, thereby reducing the processing speed of neural networks and affecting the deployment of neural networks in mobile terminals.

[0003] Quantizing a neural network is a commonly used method to reduce the size of a neural network. Quantizing a neural network means converting floating-point numbers expressed in 32-bit or 64-bit in a neural network into fixed-point numbers expressed mainly in 8-bit, and also supports fixed-point numbers with lower bits.

[0004] The existing neural network quantization method is to quantize after the neural network is trained. Although the operation is relatively simple, it usually results in a large loss of accuracy. Although the perceptual quantization method can solve the problem of accuracy loss to a certain extent, it requires an additional training process, which will increase the cost. Summary of the invention

[0005] The present invention provides a neural network adaptive quantization method, device, equipment, medium and product to solve the defects of the neural network quantization method in the prior art, such as large precision loss and increased cost, and can reduce the precision loss caused by quantization without increasing the cost.

[0006] In a first aspect, the present invention provides a neural network adaptive quantization method, comprising:

[0007] Adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusting the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network;

[0008] Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the output value of the activation function is close;

[0009] The weight values ​​and activation function output values ​​in the equivalent transformed neural network are quantized to obtain the target neural network.

[0010] According to a neural network adaptive quantization method provided by the present invention, the weight value is adjusted based on the contribution of the weight value of the neural network to the output result of the neural network; the activation function output value is adjusted based on the contribution of the activation function output value of the neural network to the output result of the neural network, including:

[0011] Determining contribution evaluation values ​​for the weight values ​​and activation function output values ​​of each layer of the neural network respectively;

[0012] Perform contribution analysis based on the determined contribution evaluation value, and determine the quantization factors of the weight value and the activation function output value respectively;

[0013] The corresponding weight value and the corresponding activation function output value are adjusted based on the determined quantization factor.

[0014] According to a neural network adaptive quantization method provided by the present invention, the contribution evaluation value is determined for each weight value and activation function output value of each layer in the neural network, including:

[0015] Determine a first mean value of the weight values ​​of each sampling unit in each layer of the neural network as a contribution evaluation value of the weight values; determine a second mean value of the activation function output value of each channel in each layer of the neural network as a contribution evaluation value of the activation function output value; and / or,

[0016] The performing contribution analysis based on the determined contribution evaluation value to respectively determine the quantization factors of the weight value and the activation function output value includes:

[0017] Determining whether the contribution evaluation value is greater than a preset threshold;

[0018] If the contribution evaluation value is greater than a preset threshold, a first quantization factor is set for the weight value or activation function output value corresponding to the contribution evaluation value;

[0019] If the contribution evaluation value is less than or equal to a preset threshold, a second quantization factor is set for the weight value or activation function output value corresponding to the contribution evaluation value; wherein the second quantization factor is less than the first quantization factor.

[0020] According to a neural network adaptive quantization method provided by the present invention, the corresponding weight value and the corresponding activation function output value are adjusted based on the determined quantization factor, including:

[0021] Based on the product of the determined quantization factor and the corresponding weight value, the weight value is updated; based on the product of the determined quantization factor and the corresponding activation function output value, the activation function output value is updated.

[0022] According to a neural network adaptive quantization method provided by the present invention, based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the activation function output value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the activation function output value is close, including:

[0023] For each layer in the adjusted neural network: determining the channel where the maximum value of the weight value and the activation function output value is located as the main channel;

[0024] Determine the scaling factors of the weight values ​​and the activation function output values ​​of the other channels except the main channel respectively;

[0025] The adjusted neural network is equivalently transformed based on the determined scaling factor.

[0026] According to a neural network adaptive quantization method provided by the present invention, the step of respectively determining the scaling factors of other channels except the main channel in the weight value and the activation function output value includes:

[0027] The ratio of the maximum weight value of the main channel to the maximum weight value of other channels except the main channel in the weight value is used as the first scaling factor of the corresponding other channels; the ratio of the maximum activation function output value of the main channel to the maximum weight value of other channels except the main channel in the activation function output value is used as the second scaling factor of the corresponding other channels; and / or,

[0028] The performing equivalent transformation on the adjusted neural network based on the determined scaling factor comprises:

[0029] Based on the product of the first scaling factor and the corresponding weight value in other channels, the distribution of the weight value channels is made close; based on the product of the second scaling factor and the corresponding activation function output value in other channels, the distribution of the activation function output value channels is made close;

[0030] The weight value in the next layer of the adjusted neural network is divided by the first scaling factor; and the activation function output value in the next layer of the adjusted neural network is divided by the second scaling factor.

[0031] In a second aspect, the present invention further provides an adaptive neural network quantization device, comprising:

[0032] A contribution analysis module, configured to adjust the weight value of the neural network based on the contribution of the weight value of the neural network to the output result of the neural network; and adjust the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network;

[0033] An equivalent transformation module is used to perform an equivalent transformation on the adjusted neural network based on the channel where the maximum value of the weight value in the adjusted neural network is located, so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, perform an equivalent transformation on the adjusted neural network, so that the distribution of each channel of the output value of the activation function is close;

[0034] The quantization processing module is used to quantize the weight values ​​and activation function output values ​​in the neural network after equivalent transformation to obtain the target neural network.

[0035] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the neural network adaptive quantization methods described above are implemented.

[0036] In a fourth aspect, the invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the neural network adaptive quantization methods described above.

[0037] In a fifth aspect, the invention further provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the neural network adaptive quantization methods described above.

[0038] The neural network adaptive quantization method, device, equipment, medium and product provided by the present invention utilize the contribution to the output result of the neural network to perform a first adaptive adjustment on the weight value and the activation function output value of the neural network, and can effectively ensure the accuracy of the neural network after quantization by enhancing the weight value and the activation function output value with a large contribution to the output result and weakening the weight value and the activation function output value with a large contribution to the output result; utilize the channel where the maximum value of the weight value and the activation function value in the adjusted neural network is located to perform a second adaptive adjustment on the weight value and the activation function output value of the neural network, and can reduce the quantization accuracy loss caused by the inconsistent distribution of each channel by making the distribution of each channel of the weight value and the activation function output value close; and no additional training process is required, which will not cause an increase in cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the neural network adaptive quantization method provided by the present invention;

[0041] Figure 2 It is a flow chart of a method for adjusting a neural network based on contribution provided by the present invention;

[0042] Figure 3 It is a flow chart of a method for equivalently transforming a neural network based on a maximum value channel provided by the present invention;

[0043] Figure 4 It is a schematic diagram of an application scenario embodiment of the neural network adaptive quantization method provided by the present invention;

[0044] Figure 5 It is a schematic diagram of the composition structure of the adaptive neural network quantization device provided by the present invention;

[0045] Figure 6 It is a schematic diagram of the composition structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Combine the following Figure 1-Figure 4 The neural network adaptive quantization method of the present invention is described.

[0048] See also Figure 1 , Figure 1 is a flow chart of the neural network adaptive quantization method provided by the present invention, Figure 1 The neural network adaptive quantization method shown can be executed by an adaptive neural network quantization device, which can be set at a server or terminal, such as a smart phone, a notebook computer, a tablet computer, a car computer, or other mobile terminal, which is not limited in the embodiment of the present invention. Figure 1 As shown, the neural network adaptive quantization method at least includes:

[0049] 101, adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusting the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network.

[0050] In an embodiment of the present invention, the neural network that performs adaptive quantization may be a neural network obtained after training the neural network. For example, the neural network may be a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), etc. The embodiment of the present invention does not limit the type of the neural network that performs adaptive quantization. The neural network that performs adaptive quantization may be a neural network for computer vision processing, or may be a neural network for natural language processing, etc. The embodiment of the present invention does not limit the application field of the neural network that performs adaptive quantization. For example, the neural network that performs adaptive quantization may be a neural network for identifying sensitive information such as CT images in the medical field or user call records. Since the training samples of such neural networks involving sensitive information are difficult to obtain, the post-training quantization method may reduce the scale of the neural network while ensuring information security, thereby obtaining a lightweight neural network.

[0051] In the embodiment of the present invention, the convolution operation and multiplication-addition operation of the weight value (weight) of the neural network and the activation function output value (input activation) of the neural network are basic operations in the neural network. Therefore, each weight value and each activation function output value in the neural network contribute to the output result of the neural network. However, since the contribution of the weight values ​​and activation function output values ​​at different positions in the neural network to the output result of the neural network is often different, for example, the deeper the number of neural network layers, the higher the abstract degree of the obtained features, the more it can reflect the essential difference, and the greater the contribution of the corresponding weight values ​​and activation function output values ​​to the output result. Therefore, by respectively determining the contribution of the weight values ​​and activation function output values ​​in the neural network to the output result of the neural network, the corresponding weight values ​​and activation function output values ​​in the neural network can be adjusted respectively to increase the gap between the weight values ​​and activation function output values ​​with large contributions and the weight values ​​and activation function output values ​​with small contributions in the adjusted neural network.

[0052] In an embodiment of the present invention, weight values ​​or activation function output values ​​in a neural network can be removed by pruning, and the accuracy of the neural network after pruning can be compared with the accuracy of the neural network before pruning to determine the contribution of the pruned weight values ​​or activation function output values ​​to the output result of the neural network.

[0053] Optionally, the weight value and activation function output value in the neural network can be adjusted by multiplying the contribution of the determined weight value or activation function output value to the output result of the neural network with the corresponding weight value or activation function output value; or, the contribution of the determined weight value or activation function output value to the output result of the neural network can be compared with a preset threshold, and the weight value or activation function output value whose contribution is less than or equal to the preset threshold is set to 0, and the weight value or activation function output value whose contribution is greater than the preset threshold is retained, so as to adjust the weight value and activation function output value in the neural network. The embodiment of the present invention does not limit the method for adjusting the weight value and activation function output value in the God General Network according to the contribution of the weight value and activation function output value to the output result of the neural network.

[0054] 102. Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the output value of the activation function is close.

[0055] In an embodiment of the present invention, after the God General Network is adjusted based on the contribution of the weight value and the activation function output value to the output result of the neural network to obtain the adjusted neural network, the channel where the maximum weight value is located in the weight values ​​of the adjusted neural network is determined, and the other channels in the weight values ​​of the neural network except the channel where the maximum weight value is located are equivalently transformed, so that the weight value distribution of the channel where the non-maximum weight value is located is close to the weight value distribution of the channel where the maximum weight value is located; and the channel where the maximum activation function output value is located in the activation function output values ​​of the adjusted neural network is determined, and the other channels in the activation function output values ​​of the neural network except the channel where the maximum activation function output value is located are equivalently transformed, so that the activation function output value distribution of the channel where the non-maximum activation function output value is located is close to the activation function output value distribution of the channel where the maximum activation function output value is located.

[0056] In an embodiment of the present invention, an equivalent transformation can be achieved by multiplying the weight value or activation function output value of the channel where the non-maximum value in the current layer in the adjusted neural network is located by a transformation factor, and dividing the weight value or activation function output value of the next layer of the current layer in the adjusted neural network by the transformation factor.

[0057] Optionally, the conversion factor of the equivalent transformation can be determined according to the maximum weight value of the channel where the maximum value is determined and the maximum weight value of other channels, and the weight value of the corresponding other channels and the weight value of the next layer are equivalently transformed; the conversion factor of the equivalent transformation can be determined according to the maximum activation function output value of the channel where the maximum value is determined and the maximum activation function output value of other channels, and the activation function output value of the corresponding other channels and the activation function output value of the next layer are equivalently transformed. Alternatively, the conversion factor of the equivalent transformation can be determined according to the maximum weight value and the average of the weight value of the channel where the maximum value is determined and the average of the maximum weight value and the weight value of other channels, and the weight value of the corresponding other channels and the weight value of the next layer are equivalently transformed; the conversion factor of the equivalent transformation can also be determined according to the maximum activation function output value and the average of the activation function output value of the channel where the maximum value is determined and the average of the maximum activation function output value and the activation function output value of other channels, and the activation function output value of the corresponding other channels and the activation function output value of the next layer are equivalently transformed. The embodiment of the present invention does not limit the method of determining the conversion factor according to the channel where the maximum value is located in the weight value and the activation function output value.

[0058] 103, quantizing the weight values ​​and activation function output values ​​in the equivalently transformed neural network to obtain a target neural network.

[0059] In an embodiment of the present invention, after performing equivalent transformation on the adjusted neural network based on the channel where the maximum values ​​of the weight values ​​and the activation function output values ​​are located to obtain the equivalent transformed neural network, the existing quantization processing method can be used to quantize the weight values ​​and the activation function output values ​​in the equivalent transformed neural network to obtain a target neural network with a reduced scale. The embodiment of the present invention does not limit the method for quantizing the weight values ​​and the activation function output values ​​in the equivalent transformed neural network.

[0060] The neural network adaptive quantization method provided by the embodiment of the present invention adjusts the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusts the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network; performs an equivalent transformation on the adjusted neural network based on the channel where the maximum value of the weight value in the adjusted neural network is located, so that the distribution of each channel of the weight value is close; performs an equivalent transformation on the adjusted neural network based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, so that the distribution of each channel of the output value of the activation function is close; performs quantization processing on the weight value and the output value of the activation function in the equivalently transformed neural network to obtain a target neural network ; Using the contribution to the output result of the neural network, the weight value and the activation function output value of the neural network are adaptively adjusted for the first time. By enhancing the weight value and the activation function output value that contribute greatly to the output result and weakening the weight value and the activation function output value that contribute greatly to the output result, the accuracy of the neural network after quantization can be effectively guaranteed; using the channel where the maximum value of the weight value and the activation function value in the adjusted neural network is located, the weight value and the activation function output value of the neural network are adaptively adjusted for the second time. By making the distribution of each channel of the weight value and the activation function output value close, the quantization accuracy loss caused by the inconsistent distribution of each channel can be reduced; and no additional training process is required, which will not increase the cost.

[0061] See also Figure 2 , Figure 2 is a flow chart of the method for adjusting the neural network based on contribution provided by the present invention, such as Figure 2 As shown, the method at least includes:

[0062] 201, determining contribution evaluation values ​​for the weight values ​​and activation function output values ​​of each layer of the neural network.

[0063] In an embodiment of the present invention, the contribution judgment value of the weight value and the activation function output value in the current layer can be determined for each layer in the neural network, and the contribution judgment value is used to determine the contribution of the weight value and the activation function output value to the output result of the neural network. Optionally, the contribution judgment value of the weight value and the activation function output value can be determined according to a pre-set algorithm, for example, the pre-set algorithm can be a mean algorithm, etc. The embodiment of the present invention does not limit the method for determining the contribution judgment value of the weight value and the activation function output value.

[0064] In some optional examples, a first mean value of the weight values ​​of each sampling unit in each layer of the neural network can be determined according to a first mean algorithm as a contribution judgment value of the weight value. For example, the first mean algorithm can be an arithmetic mean of absolute values. A second mean value of the activation function output value of each channel in each layer of the neural network can be determined according to a second mean algorithm as a contribution judgment value of the activation function output value. For example, the second mean algorithm can be an arithmetic mean.

[0065] For example, if the number of sampling units in the current layer of the neural network, including the convolution kernel, is n, and the size of the i-th sampling unit is W i ×H i ×C i The matrix, where W i is the width of sampling unit i, H i is the height of sampling unit i, C i is the number of channels of sampling unit i, then the contribution judgment value of the weight value of this layer is n, and the i-th contribution judgment value among the n contribution judgment values ​​can be calculated by calculating W in the i-th sampling unit i ×H i ×C i The arithmetic mean of the absolute values ​​of the weight values ​​is obtained.

[0066] For example, if the size of the activation function output value of the current layer in the neural network is a matrix of W'×H'×C', where W' is the width of the activation function output value, H' is the height of the activation function output value, and C' is the number of channels of the activation function output value, then the contribution judgment value of the activation function output value of this layer is C', and the j-th contribution judgment value among the C' contribution judgment values ​​can be obtained by calculating the arithmetic mean of the W'×H' activation function output values ​​in the j-th channel.

[0067] 202 , performing contribution analysis based on the determined contribution evaluation value, and determining the quantization factors of the weight value and the activation function output value respectively.

[0068] In an embodiment of the present invention, after determining the contribution judgment value of the weight value and the activation function output value of each layer in the neural network, a contribution analysis can be performed based on the contribution judgment value, and the quantization factors of the weight value and the activation function output value are determined according to the contribution analysis results, so that the quantization factors reflect the contribution of the weight value and the activation function output value to the output result of the neural network. Optionally, the contribution analysis can be performed based on a pre-set threshold, and the pre-set threshold can be set based on experience. The embodiment of the present invention does not limit the method of performing contribution analysis based on the contribution judgment value.

[0069] In some optional examples, the contribution judgment value can be compared with a preset threshold to determine whether the contribution judgment value is greater than the preset threshold; if the contribution judgment value is greater than the preset threshold, a first quantization factor can be set for the weight value or activation function output value corresponding to the contribution judgment value according to a preset rule; if the contribution judgment value is less than or equal to the preset threshold, a second quantization factor can be set for the weight value or activation function output value corresponding to the contribution judgment value according to a preset rule; wherein the second quantization factor is less than the first quantization factor, and the first quantization factor and the second quantization factor can be set based on experience.

[0070] Among them, the pre-set rules can be set according to the following principles: the larger the quantization factor, the greater the contribution to the output result, and the more important the corresponding weight value or activation function output value; the smaller the quantization factor, the smaller the contribution to the output result, and the less important the corresponding weight value or activation function output value.

[0071] For example, the pre-set rules may be: if the contribution judgment value is greater than a pre-set threshold, the weight value corresponding to the contribution judgment value or the quantization factor of the activation function output value is set to 1; if the contribution judgment value is less than or equal to the pre-set threshold, the weight value corresponding to the contribution judgment value or the quantization factor of the activation function output value is set to 0.

[0072] Alternatively, the pre-set rules may be: if the contribution evaluation value is greater than a pre-set threshold, the quantization factor of the weight value or the activation function output value corresponding to the contribution evaluation value is set to 0.7 or 0.9; if the contribution evaluation value is less than or equal to the pre-set threshold, the quantization factor of the weight value or the activation function output value corresponding to the contribution evaluation value is set to 0.3 or 0.1.

[0073] 203 , adjusting the corresponding weight value and the corresponding activation function output value based on the determined quantization factor.

[0074] In an embodiment of the present invention, after determining the quantization factors of the weight values ​​and the activation function output values ​​according to the contribution evaluation value, the corresponding weight values ​​and the activation function output values ​​can be adjusted according to the quantization factors of the weight values ​​and the activation function output values, respectively, to increase the gap between the weight values ​​and the activation function output values ​​with large contributions and the weight values ​​and the activation function output values ​​with small contributions in the adjusted neural network. Optionally, the weight values ​​and the activation function output values ​​can be adjusted by multiplying the weight values ​​and the activation function output values ​​by the corresponding quantization factors. The embodiment of the present invention does not limit the method for adjusting the weight values ​​and the activation function output values ​​according to the quantization factors.

[0075] In some optional examples, the value of the quantization factor can be between [0,1], and the weight value can be updated based on the product of the determined quantization factor and the corresponding weight value to adjust the weight value, wherein the quantization factor can be the first quantization factor or the second quantization factor; based on the product of the determined quantization factor and the corresponding activation function output value, the activation function output value is updated to adjust the activation function output value, wherein the quantization factor can be the first quantization factor or the second quantization factor.

[0076] Since, in a neural network, the weight value is usually between [-1,1], and the activation function output value is usually between [0,1], when the first quantization factor and the second quantization factor are between [0,1], and the first quantization factor is greater than the second quantization factor, the adjusted weight value and activation function output value obtained by multiplying the weight value and the activation function output value by the first quantization factor are usually significantly greater than the adjusted weight value and activation function output value obtained by multiplying the weight value and the activation function output value by the second quantization factor.

[0077] The embodiment of the present invention introduces a quantization factor by determining a contribution judgment value of the weight value and the activation function output value. According to the size of the quantization factor, the weight value and the activation function output value in the neural network are divided by importance. The weight value and the activation function output value with large contribution can be selected as the weight value and the activation function output value to be quantized. The weight value and the activation function output value with small contribution can be ignored during quantization. The accuracy of the neural network after quantization can be effectively guaranteed, and the amount of data processed by quantization can be effectively reduced, thereby improving the efficiency of the entire quantization process.

[0078] See also Figure 3 , Figure 3 is a flow chart of a method for equivalently transforming a neural network based on a maximum value channel provided by the present invention, such as Figure 3 As shown, the method includes at least:

[0079] 301, determine the channel where the maximum value of the weight value and the activation function output value is located as the main channel.

[0080] In an embodiment of the present invention, the channel where the maximum weight value and the maximum activation function output value are located in the weight values ​​and activation function output values ​​in the current layer can be determined respectively as the dominant channel (dominant channel) of the weight values ​​and activation function output values ​​of the current layer. For example, for the weight values ​​in the current layer, the maximum weight value in the weight values ​​of each channel of each sampling unit can be first determined, and then a maximum weight value of the current layer can be determined from the maximum weight values ​​corresponding to each channel of each sampling unit determined, and finally, according to the determined maximum weight value of the current layer, the channel where the maximum weight value is located is determined as the dominant channel of the weight value of the current layer.

[0081] For example, for the activation function output value in the current layer, the maximum activation function output value among the activation function output values ​​of each channel of the activation function output value can be first determined, and then a maximum activation function output value of the current layer can be determined from the maximum activation function output values ​​corresponding to the determined channels. Finally, according to the determined maximum activation function output value of the current layer, the channel where the maximum activation function output value is located is determined as the main channel of the activation function output value of the current layer.

[0082] 302, respectively determine the scaling factors of the weight values ​​and the activation function output values ​​of the other channels except the main channel.

[0083] In an embodiment of the present invention, after determining the main channels of the weight values ​​and activation function output values ​​in the current layer, the weight values ​​and activation function output values ​​in the current layer other than the main channels can be determined according to the determined main channels of the weight values ​​and activation function output values, and the scaling factors of the weight values ​​and activation function output values ​​in the current layer other than the main channels can be determined. Optionally, the scaling factors of the corresponding other channels can be determined according to the maximum value of the main channel and the maximum value of the other channels. The embodiment of the present invention does not limit the method for determining the scaling factors of the other channels other than the main channels.

[0084] For example, for the weight values ​​in the current layer, the ratio of the maximum weight value of the main channel to the maximum weight value of other channels can be calculated, and the ratio of the maximum weight value of the main channel to the maximum weight value of other channels in the weight value can be used as the first scaling factor of the corresponding other channels. For example, for the activation function output value in the current layer, the ratio of the maximum activation function output value of the main channel to the maximum activation function output value of other channels can be calculated, and the ratio of the maximum activation function output value of the main channel to the maximum activation function output value of other channels in the activation function output value can be used as the second scaling factor of the corresponding other channels.

[0085] 303 , performing equivalent transformation on the adjusted neural network based on the determined scaling factor.

[0086] In an embodiment of the present invention, after determining the scaling factors of other channels of weight values ​​and activation function output values ​​in the current layer, the weight values ​​and activation function output values ​​in the adjusted neural network can be equivalently transformed according to the scaling factors of other channels of weight values ​​and activation function output values, so that the weight value distribution of the channel where the non-maximum value in the current layer is located is close to the weight value distribution of the channel where the maximum value is located, and the activation function output value distribution of the channel where the non-maximum value is located is close to the activation function output value distribution of the channel where the maximum value is located.

[0087] Optionally, the adjusted neural network can be equivalently transformed by multiplying the weight values ​​and activation function output values ​​in other channels in the current layer by the corresponding scaling factor, and multiplying the weight values ​​and activation function output values ​​in the next layer of the current layer by the corresponding scaling factor. The embodiment of the present invention does not limit the method for equivalently transforming the adjusted neural network according to the scaling factor.

[0088] For example, for the weight values ​​in the current layer, the product of the first scaling factor of the weight values ​​and the corresponding weight values ​​in other channels can be calculated. Based on the product of the first scaling factor and the corresponding weight values ​​in other channels, the range of weight values ​​in the channels where the non-maximum values ​​are located can be expanded by using the value of the first scaling factor greater than 1, so that the distribution of weight values ​​in each channel is close; for the weight values ​​in the next layer of the current layer, the equivalent transformation of the adjusted neural network can be achieved by dividing the weight values ​​by the first scaling factor.

[0089] For example, for the activation function output value in the current layer, the product of the second scaling factor of the activation function output value and the corresponding activation function output value in other channels can be calculated. Based on the product of the second scaling factor and the corresponding activation function output value in other channels, the range of the activation function output value in the channel where the non-maximum value is located is expanded by using the value of the second scaling factor greater than 1, so that the distribution of the activation function output value of each channel is close; for the activation function output value in the next layer of the current layer, the equivalent transformation of the adjusted neural network can be achieved by dividing the activation function output value by the second scaling factor.

[0090] See also Figure 4 , Figure 4 is a schematic diagram of an application scenario embodiment of the neural network adaptive quantization method provided by the present invention, such as Figure 4As shown, the neural network in which weight and input activation are represented by 32-bit floating point numbers is adaptively quantized, including: first, analyzing the contribution of weight and input activation in the neural network, introducing an adaptive quantization factor by evaluating the contribution of weight and input activation to the output result of the neural network, adjusting the neural network by multiplying the weight and input activation in the neural network by the adaptive scaling factor, selecting the weight and input activation to be quantized, and obtaining the adjusted neural network. Then, for each layer in the adjusted neural network, find the channel corresponding to the maximum weight and input activation as the dominant channel, introduce an adaptive scaling factor for other non-dominant channels, multiply other non-dominant channels of weight and input activation by the adaptive scaling factor, make the distribution of non-dominant channels of weight and input activation close to that of dominant channels, and divide the weight and input activation in the next layer by the adaptive scaling factor to obtain the equivalently transformed neural network. Finally, quantizing the weight and input activation in the equivalently transformed neural network to obtain the target neural network in which weight and input activation are represented by 8-bit fixed-point numbers after quantization. Table 1 takes an 18-layer deep residual network (ResNet) and MobileNet as examples, and lists the network accuracy when 32-bit floating point numbers are used to represent weights and input activations, the network accuracy after quantizing the weights and input activations represented by 32-bit floating point numbers using a conventional perchannel quantization method, and the network accuracy after quantizing the weights and input activations represented by 32-bit floating point numbers using an adaptive quantization method according to an embodiment of the present invention.

[0091] Table 1

[0092]

[0093]

[0094] It can be seen from the experimental data in Table 1 that, compared with the conventional quantization method, the accuracy of the network obtained by quantizing the neural network using the adaptive quantization method of the embodiment of the present invention is closer to the accuracy of the network before quantization. Therefore, the adaptive quantization method of the embodiment of the present invention is used to quantize the neural network without causing a large loss of accuracy.

[0095] The adaptive neural network quantization device provided by the present invention is described below. The adaptive neural network quantization device described below and the neural network adaptive quantization method described above can be referenced to each other.

[0096] See also Figure 5 , Figure 5 is a schematic diagram of the composition structure of the adaptive neural network quantization device provided by the present invention, Figure 5 The adaptive neural network quantization device shown can be used to perform Figure 1 Neural network adaptive quantization method, such as Figure 5 As shown, the adaptive neural network quantization device at least includes:

[0097] The contribution analysis module 510 is used to adjust the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; and adjust the activation function output value based on the contribution of the activation function output value of the neural network to the output result of the neural network.

[0098] The equivalent transformation module 520 is used to perform an equivalent transformation on the adjusted neural network based on the channel where the maximum value of the weight value in the adjusted neural network is located, so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, perform an equivalent transformation on the adjusted neural network, so that the distribution of each channel of the output value of the activation function is close.

[0099] The quantization processing module 530 is used to quantize the weight values ​​and activation function output values ​​in the equivalent transformed neural network to obtain the target neural network.

[0100] Optionally, the contribution analysis module 510 includes:

[0101] The evaluation value calculation unit is used to determine the contribution evaluation value for each layer of the neural network weight value and activation function output value.

[0102] The contribution analysis unit is used to perform contribution analysis based on the determined contribution evaluation value, and determine the quantization factors of the weight value and the activation function output value respectively.

[0103] The quantization factor adjustment unit is used to adjust the corresponding weight value and the corresponding activation function output value respectively based on the determined quantization factor.

[0104] Optionally, the evaluation value calculation unit is used to determine a first mean value of the weight values ​​of each sampling unit in each layer of the neural network as a contribution evaluation value of the weight value; determine a second mean value of the activation function output value of each channel in each layer of the neural network as a contribution evaluation value of the activation function output value; and / or,

[0105] A contribution analysis unit is used to determine whether a contribution evaluation value is greater than a preset threshold; if the contribution evaluation value is greater than the preset threshold, a first quantization factor is set for the weight value or activation function output value corresponding to the contribution evaluation value; if the contribution evaluation value is less than or equal to the preset threshold, a second quantization factor is set for the weight value or activation function output value corresponding to the contribution evaluation value; wherein the second quantization factor is less than the first quantization factor.

[0106] Optionally, the quantization factor adjustment unit is used to update the weight value based on the product of the determined quantization factor and the corresponding weight value; and to update the activation function output value based on the product of the determined quantization factor and the corresponding activation function output value.

[0107] Optionally, the equivalent transformation module 520 includes:

[0108] The main channel selection unit is used to determine, for each layer in the adjusted neural network, a channel where the maximum value of the weight value and the activation function output value is located as the main channel.

[0109] The scaling factor calculation unit is used to determine the scaling factors of other channels except the main channel in the weight value and the activation function output value for each layer in the adjusted neural network.

[0110] The equivalent transformation unit is used to perform an equivalent transformation on the adjusted neural network based on the determined scaling factor for each layer in the adjusted neural network.

[0111] Optionally, the scaling factor calculation unit is used to use the ratio of the maximum weight value of the main channel to the maximum weight value of other channels except the main channel in the weight value as the first scaling factor of the corresponding other channels; use the ratio of the maximum activation function output value of the main channel to the maximum weight value of other channels except the main channel in the activation function output value as the second scaling factor of the corresponding other channels; and / or,

[0112] The equivalent transformation unit is used to make the distribution of weight values ​​in each channel close based on the product of the first scaling factor and the corresponding weight values ​​in other channels; make the distribution of activation function output values ​​in each channel close based on the product of the second scaling factor and the corresponding activation function output values ​​in other channels; divide the weight values ​​in the next layer of the adjusted neural network by the first scaling factor; and divide the activation function output values ​​in the next layer of the adjusted neural network by the second scaling factor.

[0113] The present invention also provides a neural network obtained by quantizing a neural network based on the neural network adaptive quantization method provided by the above methods. Optionally, the neural network can be an image recognition neural network, or a text classification neural network, or a speech recognition neural network, etc., which is not limited in the embodiment of the present invention.

[0114] The present invention also provides a CT image recognition method, which inputs a CT image into a neural network for processing, and obtains information about cancer tissue in the CT image output by the neural network. Optionally, the information about cancer tissue in the CT image may include properties of the cancer tissue, such as benign or malignant. The neural network is a neural network obtained by processing the neural network using the neural network adaptive quantization method provided by the above methods.

[0115] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the neural network adaptive quantization method, which includes:

[0116] Adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusting the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network;

[0117] Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the output value of the activation function is close;

[0118] The weight values ​​and activation function output values ​​in the equivalent transformed neural network are quantized to obtain the target neural network.

[0119] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0120] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the neural network adaptive quantization method provided by the above methods, the method comprising:

[0121] Adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusting the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network;

[0122] Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the output value of the activation function is close;

[0123] The weight values ​​and activation function output values ​​in the equivalent transformed neural network are quantized to obtain the target neural network.

[0124] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for performing the above-mentioned neural network adaptive quantization method is implemented, and the method comprises:

[0125] Adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; adjusting the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network;

[0126] Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the output value of the activation function is close;

[0127] The weight values ​​and activation function output values ​​in the equivalent transformed neural network are quantized to obtain the target neural network.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network adaptive quantization method, characterized in that: include: Adjusting the weight value of the neural network based on the contribution of the weight value of the neural network to the output result of the neural network; Adjusting the activation function output value based on the contribution of the activation function output value of the neural network to the output result of the neural network; wherein the neural network is an image recognition neural network, a text classification neural network or a speech recognition neural network; Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the output value of the activation function is close; Quantize the weight values ​​and activation function output values ​​in the equivalently transformed neural network to obtain the target neural network; The step of adjusting the weight value based on the contribution of the weight value of the neural network to the output result of the neural network; and adjusting the output value of the activation function based on the contribution of the output value of the activation function of the neural network to the output result of the neural network, comprises: Determining contribution evaluation values ​​for the weight values ​​and activation function output values ​​of each layer of the neural network respectively; Perform contribution analysis based on the determined contribution evaluation value, and determine the quantization factors of the weight value and the activation function output value respectively; Adjusting the corresponding weight value and the corresponding activation function output value respectively based on the determined quantization factor; Determining contribution evaluation values ​​for weight values ​​and activation function output values ​​of each layer in the neural network respectively includes: Determine a first mean value of the weight values ​​of each sampling unit in each layer of the neural network as a contribution evaluation value of the weight values; determine a second mean value of the activation function output value of each channel in each layer of the neural network as a contribution evaluation value of the activation function output value; The performing contribution analysis based on the determined contribution evaluation value to respectively determine the quantization factors of the weight value and the activation function output value includes: Determining whether the contribution evaluation value is greater than a preset threshold; If the contribution evaluation value is greater than a preset threshold, a first quantization factor is set for the weight value or activation function output value corresponding to the contribution evaluation value; If the contribution evaluation value is less than or equal to a preset threshold, a second quantization factor is set for the weight value or activation function output value corresponding to the contribution evaluation value; wherein the second quantization factor is less than the first quantization factor.

2. The neural network adaptive quantization method according to claim 1, characterized in that: The adjusting the corresponding weight value and the corresponding activation function output value based on the determined quantization factor respectively includes: Based on the product of the determined quantization factor and the corresponding weight value, the weight value is updated; based on the product of the determined quantization factor and the corresponding activation function output value, the activation function output value is updated.

3. The neural network adaptive quantization method according to claim 1 or 2, characterized in that: Based on the channel where the maximum value of the weight value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the activation function output value in the adjusted neural network is located, the adjusted neural network is equivalently transformed so that the distribution of each channel of the activation function output value is close, including: For each layer in the adjusted neural network: determining the channel where the maximum value of the weight value and the activation function output value is located as the main channel; Determine the scaling factors of the weight values ​​and the activation function output values ​​of the other channels except the main channel respectively; The adjusted neural network is equivalently transformed based on the determined scaling factor.

4. The neural network adaptive quantization method according to claim 3, characterized in that: The step of respectively determining the scaling factors of the weight values ​​and the activation function output values ​​of the other channels except the main channel comprises: The ratio of the maximum weight value of the main channel to the maximum weight value of other channels except the main channel in the weight value is used as the first scaling factor of the corresponding other channels; the ratio of the maximum activation function output value of the main channel to the maximum weight value of other channels except the main channel in the activation function output value is used as the second scaling factor of the corresponding other channels; and / or, The performing equivalent transformation on the adjusted neural network based on the determined scaling factor comprises: Based on the product of the first scaling factor and the corresponding weight value in other channels, the distribution of the weight value channels is made close; based on the product of the second scaling factor and the corresponding activation function output value in other channels, the distribution of the activation function output value channels is made close; The weight value in the next layer of the adjusted neural network is divided by the first scaling factor; and the activation function output value in the next layer of the adjusted neural network is divided by the second scaling factor.

5. An adaptive neural network quantization device based on the neural network adaptive quantization method according to any one of claims 1 to 4, characterized in that: include: A contribution analysis module, configured to adjust the weight value of the neural network based on the contribution of the weight value of the neural network to the output result of the neural network; Adjusting the activation function output value based on the contribution of the activation function output value of the neural network to the output result of the neural network; wherein the neural network is an image recognition neural network, a text classification neural network or a speech recognition neural network; An equivalent transformation module is used to perform an equivalent transformation on the adjusted neural network based on the channel where the maximum value of the weight value in the adjusted neural network is located, so that the distribution of each channel of the weight value is close; based on the channel where the maximum value of the output value of the activation function in the adjusted neural network is located, perform an equivalent transformation on the adjusted neural network, so that the distribution of each channel of the output value of the activation function is close; The quantization processing module is used to quantize the weight values ​​and activation function output values ​​in the neural network after equivalent transformation to obtain the target neural network.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the neural network adaptive quantization method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the neural network adaptive quantization method according to any one of claims 1 to 4 are implemented.

8. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the neural network adaptive quantization method according to any one of claims 1 to 4 are implemented.

Images