Deep visual feature quantization method and device based on feature channel bit allocation

Abstract

The present invention discloses a method and device for quantizing deep visual features based on feature channel bit allocation, the method comprising the following steps: obtaining an original intermediate layer feature vector to be processed; pre-allocating channel bits for the original intermediate layer feature vector to be processed using an offline iterative method; after pre-allocating quantization intervals using the offline iterative method in step S2, reallocating bits for the remaining unallocated quantization intervals according to the statistical characteristics of the feature vector; outputting the quantized intermediate layer feature vector; the present invention can quantize the intermediate layer feature vector; has strong adaptability to different task requirements and different deployment environment requirements, and achieves a balance between computational complexity and network output error; and can obtain the minimum network output error under the current evaluation index.

Description

Technical Field

[0001] The present invention relates to the technical field of intermediate layer feature compression and transmission, and in particular to a method and device for quantizing deep visual features based on feature channel bit allocation. Background Art

[0002] One of the key technologies for compressed transmission of intermediate layer features is quantization. Quantization refers to the process of approximating the continuous value of a signal or a large number of possible discrete values ​​to a finite number of or fewer discrete values. Deep learning models are usually trained with high bit depth floating point numbers to ensure high accuracy of the back propagation process. Therefore, the intermediate deep learning features are usually also in high bit depth floating point format. However, in neural network reasoning, small errors in intermediate feature elements usually do not affect the final output, which can be achieved by applying quantization technology to reduce the amount of feature data. Most of the existing technologies currently only use uniform quantizers or logarithmic quantizers to simply reduce the amount of feature data, and do not pay enough attention to the quantizer optimization for network output errors and the corresponding rate distortion optimization; one technology designs an entropy constrained quantization method and cooperates with entropy coding in the compression stage to optimize the compression rate as a whole, but this technology does not optimize the task performance of a specific deep learning network; another technology implements a quantization method for bit allocation between several feature vectors in different layers of a deep learning model or even several different deep learning models, but this technology does not explore bit allocation in a single feature vector.

[0003] In the related research on neural network model compression, some quantization techniques have also been proposed. However, most of them rely on the subsequent retraining of deep learning network models to compensate for the information loss caused by the quantization stage. In terms of quantization technology, they simply design quantization methods to optimize the quantization error caused by quantization of feature vectors. At the same time, these quantization techniques do not consider the complexity of calculation. This is not practical for real-time feature encoding in data communication scenarios.

[0004] In summary, the existing quantization technology solutions have the following shortcomings: First, the quantization bits are not allocated according to the signal error output by the deep learning neural network, and the optimal bit allocation method is not found to minimize the output distortion of the overall network. Secondly, the statistical information of the feature vectors of the intermediate layers of the deep learning network model itself, such as the mean standard deviation, and the correlation between the statistical information, is not analyzed. The quantization technology does not establish a certain connection with the intermediate layer feature vectors to be quantized. Finally, it is no longer possible to adjust according to the task performance and deployment environment requirements, and it is impossible to achieve a balance between quantization performance and computing load. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects in the prior art and provide a deep visual feature quantization method and device based on feature channel bit allocation, which can quantize the intermediate layer feature vector; have strong adaptability to different task requirements and different deployment environment requirements, and at the same time achieve a balance between computational complexity and network output error; and can obtain the minimum network output error under the current evaluation index.

[0006] To achieve the above object, the present invention provides a deep visual feature quantization method based on feature channel bit allocation, comprising the following steps:

[0007] Step S1: Obtain the original intermediate layer feature vector to be processed;

[0008] Step S2: pre-allocating channel bits of the original intermediate layer feature vector to be processed using an offline iterative method;

[0009] Step S3: after the quantization intervals are pre-allocated by the offline iteration method in step S2, the remaining unallocated quantization intervals are reallocated according to the statistical characteristics of the feature vectors;

[0010] Step S4: Output the quantized intermediate layer feature vector.

[0011] Preferably, the method further includes step S5: performing online iterative allocation on the unallocated quantization intervals reserved in step S3, and outputting the quantized intermediate layer feature vector after one or more optimizations.

[0012] Preferably, pre-allocation preparation is also included; before performing feature vector bit allocation based on feature channels, feature extraction is required according to the current deep learning network model and the specific position in the network model, and pre-processing preparation for quantization bit allocation is performed on the required data set.

[0013] Preferably, the pre-allocation preparation comprises the following steps:

[0014] Step S61: On the currently selected deep learning network model, a certain number of samples are used in the selected data set to perform online iterative calculations on the current layer;

[0015] Step S62: In each iterative calculation, one feature channel is selected in the order of the feature channels of the intermediate layer feature vector and quantized according to the number of quantization intervals that have been allocated, while the number of quantization intervals allocated to other feature channels in the previous iterative calculation remains unchanged and is not quantized;

[0016] Step S63: Send the feature with only one channel quantized into the deep learning network for inference to obtain the network output error, and repeat the above process until the network output error corresponding to each channel quantized is obtained by statistics;

[0017] Step S64: Assign a new unassigned quantization interval to the feature channel that causes the maximum network output error; repeat the above process until the sum of the number of quantization intervals for each feature channel reaches C×2 R This constraint condition, where C represents the number of feature channels and R represents the number of quantization bits.

[0018] Preferably, the pre-allocation will obtain a quantization interval allocation table of how many quantization intervals should be allocated to each feature channel of the intermediate layer feature vector in a statistical sense on the current deep learning network model; the table is saved offline on the front-end device, and the quantization interval allocation table can be directly called in subsequent operations to perform offline quantization interval allocation for the intermediate layer features; the allocation of the quantization interval is a rough allocation, and there are still a certain number of quantization intervals that are unallocated.

[0019] Preferably, step S3 includes the following steps:

[0020] Step S31: The feature channel mean of the intermediate layer feature can indicate the priority relationship of the quantization interval allocation of different channels of the intermediate layer feature; the quantization interval allocation priority expression based on the feature channel mean is defined as follows:

[0021]

[0022] Among them, F i represents the i-th channel of the intermediate layer feature, μ(·) represents the averaging operation;

[0023] Step S32: The eigenvalue distribution in the intermediate layer feature channel with a large standard deviation is more discrete than that in the channel with a small standard deviation, and more quantization intervals are required to represent a larger quantization range to ensure the accuracy after quantization; therefore, the quantization interval allocation priority expression based on the standard deviation of the feature channel can be defined as follows:

[0024]

[0025]

[0026] Among them, F irepresents the i-th channel of the intermediate layer feature, μ(·) represents the standard deviation operation, and σ(·) represents the normalization operation; Formula 2 normalizes the standard deviation of the feature channel, and normalization is used to eliminate the influence of the mean value on the standard deviation on the size of the standard deviation, so that the standard deviations of different feature channels can be compared in priority; Formula 3 calculates the quantization interval allocation priority based on the standard deviation;

[0027] Finally, the quantization interval allocation priority of the i-th channel is defined as:

[0028]

[0029] Among them; α and β are empirical values;

[0030] The unallocated quantization intervals in step S2 are allocated according to the calculated feature channel quantization interval allocation priority; at the same time, in this step, some quantization intervals can be reserved and not allocated according to specific task requirements and deployment environment conditions, and these reserved quantization intervals will be allocated to the feature channel in step S5.

[0031] Preferably, step S5 includes: the specific allocation process is equivalent to the online iterative algorithm process in step S2, and only the feature channel quantization interval allocation table needs to be changed from the initial all-1 to the value after the end of step S3; the more quantization intervals reserved in step S3, the smaller the network output error that can be obtained after online iteration in this step, but the computational cost will also increase accordingly.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the deep visual feature quantization method based on feature channel bit allocation as described above are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The method provided by the present invention allocates different numbers of quantization intervals to different channels of the intermediate layer feature vectors of the deep learning network model through iterative calculation and combining the statistical characteristics of the intermediate layer features of the deep learning network model itself, so as to quantize the intermediate layer feature vectors; when allocating quantization intervals for the intermediate layer feature vectors, the number of iterations on the selective control line is used to have strong adaptability to different task requirements and different deployment environment requirements, and at the same time achieve a balance between computational complexity and network output error; through the new quantization method proposed by the present invention, the minimum network output error under the current evaluation index can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a schematic diagram of the steps of a deep visual feature quantization method based on feature channel bit allocation provided by the present invention;

[0037] Figure 2 It is a schematic diagram of the online iterative algorithm process provided by the present invention;

[0038] Figure 3 It is a schematic diagram of the relationship between the average value of the characteristic channel and the number of quantization intervals provided by the present invention;

[0039] Figure 4 It is a schematic diagram of the relationship between the characteristic channel standard deviation and the number of quantization intervals provided by the present invention. DETAILED DESCRIPTION

[0040] The technical scheme in this embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in this embodiment of the present invention. Obviously, the described embodiment is one embodiment of the present invention, not all embodiments of the present invention. Based on this embodiment of the present invention, all other embodiments of the present invention obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1

[0042] Please refer to Figures 1 to 4 , Embodiment 1 of the present invention provides a deep visual feature quantization method based on feature channel bit allocation.

[0043] like Figure 1 As shown, the present invention provides a deep visual feature quantization method based on feature channel bit allocation, and the quantization method is mainly divided into the following five steps.

[0044] Step S1: Obtain the original intermediate layer feature vector to be processed;

[0045] Step S2: pre-allocating channel bits of the original intermediate layer feature vector to be processed using an offline iterative method;

[0046] Step S3: after the quantization intervals are pre-allocated by the offline iteration method in step S2, the remaining unallocated quantization intervals are reallocated according to the statistical characteristics of the feature vectors;

[0047] Step S4: Output the quantized intermediate layer feature vector.

[0048] Step S5: Perform online iterative allocation on the unallocated quantization intervals reserved in step S3, and after one or more optimizations, output the quantized intermediate layer feature vector.

[0049] The following is a detailed description of each step process.

[0050] (1) Figure 1 The step S2: uses an offline iterative method to pre-allocate channel bits for the original intermediate layer feature vector to be processed; specifically, before performing feature vector bit allocation based on feature channels, it is necessary to perform feature extraction based on the current deep learning network model and the specific position in the network model, and perform pre-processing preparation for quantization bit allocation on the required data set.

[0051] First, on the currently selected deep learning network model, a certain number of samples are used in the selected data set to perform online iterative calculations on the current layer.

[0052] Secondly, in each iterative calculation, one feature channel will be selected according to the order of the feature channels of the intermediate layer feature vector and quantized according to the number of quantization intervals that have been allocated, while other feature channels will remain unchanged according to the number of quantization intervals allocated in the last iterative calculation and will not be quantized.

[0053] Again, the feature with only one channel being quantized is sent to the deep learning network for inference to obtain the network output error. The above process is repeated until the network output error corresponding to the quantization of each channel is statistically obtained.

[0054] Finally, a new unassigned quantization interval is assigned to the feature channel that causes the largest network output error. The above process is repeated until the sum of the number of quantization intervals for each feature channel reaches C×2 R This constraint condition, where C represents the number of feature channels and R represents the number of quantization bits.

[0055] The pre-allocation will obtain a quantization interval allocation table of how many quantization intervals should be allocated to each feature channel of the intermediate layer feature vector in a statistical sense on the current deep learning network model; the table is saved offline on the front-end device, and the quantization interval allocation table can be directly called in subsequent operations to perform offline quantization interval allocation for the intermediate layer features; due to rounding down and not targeting specific feature vectors, the allocation of the quantization interval is a rough allocation, and there are still a certain number of quantization intervals in an unallocated state; these unallocated quantization intervals will be allocated in step S3 according to the statistical properties of the feature vector itself.

[0056] Further, such as Figure 2 As shown, it is a schematic diagram of the algorithm flow of online iteration.

[0057] (2) After the quantization intervals are pre-allocated by the offline iteration method in step S2, the remaining unallocated quantization intervals are reallocated according to the correlation of the statistical information of the features themselves. The step S3 establishes the quantization interval allocation priority by analyzing the relationship between the feature channel average value and standard deviation and the number of quantization intervals, and then allocates them.

[0058] By counting the relationship between the number of quantization intervals iteratively allocated on multiple sample lines and the mean of the feature channel, it can be seen that as the mean of the feature channel of the intermediate layer increases, the number of quantization intervals allocated to the channel will also increase, showing a linear positive correlation as a whole, such as Figure 3 shown.

[0059] Therefore, it can be considered that the average value of the feature channel of the intermediate layer feature can indicate the priority relationship between the different channels of the intermediate layer feature for the quantization interval allocation. For this, the quantization interval allocation priority expression based on the feature channel average is defined as follows:

[0060]

[0061] Among them, F i represents the i-th channel of the intermediate layer feature, μ(·) represents the averaging operation;

[0062] By counting the relationship between the number of quantization intervals iteratively allocated on multiple sample lines and the standard deviation of the feature channel, it can be seen that the feature channel with a larger standard deviation tends to be allocated fewer quantization intervals, and the overall relationship presents a negative exponential relationship, such as Figure 4 shown.

[0063] The reason is that the eigenvalue distribution in the intermediate layer feature channel with a large standard deviation is more discrete than that in the channel with a small standard deviation, and more quantization intervals are needed to represent a larger quantization range to ensure the accuracy after quantization. Therefore, the quantization interval allocation priority expression based on the standard deviation of the feature channel can be defined as follows:

[0064]

[0065]

[0066] Among them, F i represents the i-th channel of the intermediate layer feature, μ(·) represents the standard deviation operation, and σ(·) represents the normalization operation; Formula 2 normalizes the standard deviation of the feature channel, and normalization is used to eliminate the influence of the mean value on the standard deviation on the size of the standard deviation, so that the standard deviations of different feature channels can be compared in priority; Formula 3 calculates the quantization interval allocation priority based on the standard deviation;

[0067] Finally, the quantization interval allocation priority of the i-th channel is defined as:

[0068]

[0069] Wherein, α and β are empirical values; in the first embodiment, α and β are 0.9 and 0.1 respectively.

[0070] Then, the unallocated quantization intervals in step S2 are allocated according to the calculated feature channel quantization interval allocation priority; at the same time, in this step, some quantization intervals can be reserved and not allocated according to specific task requirements and deployment environment conditions, and these reserved quantization intervals will be allocated to the feature channel in step S5.

[0071] (3) In the calculations in step S2 and step S3, except for the pre-allocation part in step S2, which needs to be calculated in advance and uses the reasoning process of the deep learning network, the remaining operations do not introduce the reasoning process of the deep learning network, so the computational complexity is very low and the calculation can be completed quickly. At the same time, since the statistical correlation of the intermediate layer feature vector used in the bit allocation in step S3 cannot perfectly allocate the remaining quantization intervals, it can only lead to a suboptimal solution. In order to approach the optimal solution of a bit allocation scheme, in step S5, it is possible to select the unallocated quantization intervals reserved in step S3 for online iterative allocation. The specific allocation process is the same as the online iterative algorithm process in step S2, and only the feature channel quantization interval allocation table needs to be changed from the initial all 1 to the value after the end of step S3. In theory, the more quantization intervals reserved in step S3, the smaller the network output error that can be obtained after online iteration in this step, but the computational cost will also increase accordingly. This step provides an option to balance bit allocation performance and computational complexity.

[0072] After the above steps S2, S3 and S5, the quantization technology designed by the present invention will approach the network output error to the minimum value.

[0073] Embodiment 2

[0074] Embodiment 2 of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the deep visual feature quantization method based on feature channel bit allocation as described in Embodiment 1 are implemented.

[0075] Embodiment 3

[0076] The quantization technology proposed in this paper is experimented on the intermediate layer features of two common deep learning network models, VGGNet and ResNet, on the ImageNet dataset.

[0077] The ImageNet dataset consists of 1.2 million images in 1,000 categories. Deep learning network models generally choose to extract image features on this dataset because their features can be considered universal. The VGGNet model was proposed in 2014. This deep learning network model has achieved a good balance between performance and computational complexity and is currently widely used in image feature extraction for various computer vision tasks. The ResNet model was proposed in 2015. This model has a unique structure that allows it to have a very deep network structure while maintaining a low level of computational complexity. It is currently also widely used in image feature extraction for various computer vision tasks.

[0078] The quantization technology proposed in the present invention is compared with the uniform quantization technology, logarithmic quantization technology and DoReFa quantization technology which are also applied to the quantization of a single feature vector. The comparison results are shown in the following table:

[0079] Table 1: Comparison of network output errors of ResNet18

[0080]

[0081] Table 2: Comparison of network output errors of ResNet34

[0082]

[0083] Table 3: Comparison of network output errors of ResNet50

[0084]

[0085] Table 4: Comparison of network output errors of VGGNet16

[0086]

[0087] Since logarithmic quantization is a non-uniform quantization interval, while uniform quantization and DoReFa quantization are equally spaced quantization intervals, they can only be compared separately. The comparison results are as follows:

[0088] Table 5: Comparison of network output errors of ResNet18

[0089]

[0090] Among them, Ours(0iter) represents the network output error obtained without going through the quantization process in step S5, Ours(10iter), Ours(20iter), Ours(30iter), and Ours(40iter) respectively represent the network output errors obtained by retaining 10, 20, 30, and 40 quantization intervals in step S3 and performing bit allocation in step S5.

[0091] From the above comparison results, it can be seen that the quantization technology proposed by the present invention is obviously superior to the existing quantization technology also applied to the quantization of a single feature vector. At the same time, it can be adjusted according to task requirements and environmental conditions, which is more conducive to practical applications.

[0092] Embodiment 4

[0093] The technology involved in the present invention is deployed on mobile phones or other smart mobile devices. The front-end part of the deep learning network model is deployed on front-end devices such as mobile phones, and the back-end part of the deep learning network model is deployed on cloud servers. Front-end devices such as mobile phones obtain original image or video data through sensors such as cameras, and obtain intermediate layer feature vectors through inference calculations of the front-end part of the deep learning network. The quantization method based on feature channel bit allocation proposed in the present invention is applied to the obtained feature vector, and the intermediate layer features are quantized and then encoded and sent to the cloud server. The cloud server decodes the received bit stream to obtain the quantized intermediate layer features, and then applies the corresponding dequantization method to obtain the dequantized features, which are finally sent to the back-end network deployed on the cloud server for inference and analysis and processing of subsequent tasks.

[0094] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A deep visual feature quantization method based on feature channel bit allocation, characterized by: The following steps are involved: Step S1: Obtain the original intermediate layer feature vector to be processed; Step S2: pre-allocating channel bits of the original intermediate layer feature vector to be processed using an offline iterative method; Step S3: after the quantization intervals are pre-allocated by the offline iteration method in step S2, the remaining unallocated quantization intervals are reallocated according to the statistical characteristics of the feature vectors; Step S4: output the quantized intermediate layer feature vector; It also includes pre-allocation preparation; before performing feature vector bit allocation based on feature channels, it is necessary to perform feature extraction based on the current deep learning network model and the specific position in the network model, and perform pre-processing preparation for quantization bit allocation on the required data set; The pre-allocation preparation comprises the following steps: Step S61: On the currently selected deep learning network model, a certain number of samples are used in the selected data set to perform online iterative calculations on the current layer; Step S62: In each iterative calculation, one feature channel is selected in the order of the feature channels of the intermediate layer feature vector and quantized according to the number of quantization intervals that have been allocated, while the number of quantization intervals allocated to other feature channels in the previous iterative calculation remains unchanged and is not quantized; Step S63: Send the feature with only one channel quantized into the deep learning network for inference to obtain the network output error, and repeat the above process until the network output error corresponding to each channel quantized is obtained by statistics; Step S64: Assign a new unassigned quantization interval to the feature channel that causes the maximum network output error; repeat the above process until the sum of the number of quantization intervals for each feature channel reaches C×2 R This constraint, where C represents the number of feature channels and R represents the number of quantization bits; The step S3 includes the following steps: Step S31: The feature channel mean of the intermediate layer feature indicates the priority relationship of the quantization interval allocation of different channels of the intermediate layer feature; the quantization interval allocation priority expression based on the feature channel mean is defined as follows: Among them, F i represents the i-th channel of the intermediate layer feature, μ(·) represents the averaging operation; Step S32: The eigenvalue distribution in the intermediate layer feature channel with a large standard deviation is more discrete than that in the channel with a small standard deviation, and more quantization intervals are required to represent a larger quantization range to ensure the accuracy after quantization; therefore, the quantization interval allocation priority expression based on the standard deviation of the feature channel is defined as follows: Among them, F i represents the i-th channel of the intermediate layer feature, μ(·) represents the standard deviation operation, and σ(·) represents the normalization operation; Formula 2 normalizes the standard deviation of the feature channel, and normalization is used to eliminate the influence of the mean value on the standard deviation on the size of the standard deviation, so that the standard deviations of different feature channels can be compared in priority; Formula 3 calculates the quantization interval allocation priority based on the standard deviation; Finally, the quantization interval allocation priority of the i-th channel is defined as: Among them; α and β are empirical values; The unallocated quantization intervals in step S2 are allocated according to the calculated feature channel quantization interval allocation priority; at the same time, in this step, some quantization intervals are reserved and not allocated according to specific task requirements and deployment environment conditions, and these reserved quantization intervals will be allocated to the feature channel in step S5; The depth visual feature is extracted through image features.

2. The method for quantizing deep visual features based on feature channel bit allocation according to claim 1, characterized in that: The method further includes step S5: performing online iterative allocation on the unallocated quantization intervals reserved in step S3, and outputting a quantized intermediate layer feature vector after one or more optimizations.

3. The method for quantizing deep visual features based on feature channel bit allocation according to claim 1, characterized in that: The pre-allocation will obtain a quantization interval allocation table of how many quantization intervals should be allocated to each feature channel of the intermediate layer feature vector in a statistical sense on the current deep learning network model; the table is saved offline on the front-end device, and the quantization interval allocation table is directly called in subsequent operations to perform offline quantization interval allocation on the intermediate layer features; the allocation of the quantization interval is a rough allocation, and there are still a certain number of quantization intervals in an unallocated state.

4. The method for quantizing deep visual features based on feature channel bit allocation according to claim 1, characterized in that: The step S5 includes: the specific allocation process is equivalent to the online iterative algorithm process in step S2, and only the feature channel quantization interval allocation table needs to be changed from the initial all 1 to the value after the end of step S3; the more quantization intervals reserved in step S3, the smaller the network output error obtained after the online iteration in this step, but the computational cost will also increase accordingly.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the deep visual feature quantization method based on feature channel bit allocation as described in any one of claims 1 to 4 are implemented.

Images