Neural network quantization bit width determination method and program product

Abstract

The invention discloses a neural network quantization bit width determination method and a program product, and the method comprises the steps: obtaining a weight matrix of a plurality of to-be-quantized network layers in a to-be-quantized neural network, and determining the quantization sensitivity corresponding to each to-be-quantized network layer; sorting the plurality of to-be-quantized network layers according to the magnitude sequence of the quantization sensitivities, dividing the sorted plurality of to-be-quantized network layers into a plurality of layer group combinations according to the number of types of quantization bit widths, each layer group combination comprising a plurality of network layer groups; according to the quantization sensitivity of the to-be-quantized network layer in the layer group combination, respectively allocating a quantization bit width to each network layer group in the plurality of layer group combinations, so as to obtain a plurality of quantization bit width combinations corresponding to the to-be-quantized neural network; and determining a target quantization bit width combination from a plurality of quantization bit width combinations according to the reasoning accuracy of the to-be-quantized neural network under each quantization bit width, so as to improve the efficiency of determining the quantization bit width of the neural network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and program product for determining the quantization bit width of a neural network. Background Technology

[0002] Neural networks possess powerful feature extraction capabilities, enabling them to accurately fit complex functional relationships. With the continuous increase in model parameters, network depth, and computational complexity, their learning capabilities are further enhanced, and their inference accuracy across various tasks steadily improves. However, their massive computational demands and memory consumption have become key bottlenecks restricting practical deployment. Currently, quantization bit-width compression techniques are often used to reduce the computational load and storage requirements of models, alleviating the challenges of neural network deployment.

[0003] In related technologies, determining the quantization bit width of a neural network often relies on an traversal search strategy. This involves first assigning multiple candidate quantization bit widths to the weights of each layer, then combining these candidate bit widths to form multiple quantization bit width combinations. Each combination is then quantized individually, and inference verification is performed on the target task based on the quantized network until a quantization bit width combination that meets the inference accuracy requirements is determined. This requires verifying a massive number of bit width combinations one by one, resulting in huge computational overhead and a time-consuming and inefficient quantization bit width determination process, making it unsuitable for the quantization needs of large-scale neural networks. Therefore, a quantization bit width search method is urgently needed to improve the efficiency of quantization bit width determination. Summary of the Invention

[0004] This invention provides a method and program product for determining the quantization bit width of a neural network, in order to solve the problem that related technologies that determine the quantization bit width of a neural network through traversal search strategies have large computational overhead and low efficiency.

[0005] According to one aspect of the present invention, a method for determining the quantization bit width of a neural network is provided, the method comprising:

[0006] Obtain the weight matrices of multiple network layers to be quantized in the neural network to be quantized, and determine the quantization sensitivity of each network layer to be quantized based on the weight matrix of each network layer to be quantized, wherein the neural network to be quantized is a neural network trained based on a target task;

[0007] The multiple network layers to be quantized are sorted according to the order of their quantization sensitivity. The sorted multiple network layers to be quantized are divided into multiple layer group combinations according to the number of quantization bit widths. Each layer group combination includes multiple network layer groups.

[0008] According to the quantization sensitivity of the network layer to be quantized in the layer group combination, the quantization bit width is assigned to each network layer group in the multiple layer group combinations to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized, wherein the quantization bit widths corresponding to multiple network layers to be quantized in the network layer group are the same.

[0009] Based on the inference accuracy of the neural network to be quantized under each of the quantization bit width combinations, the target quantization bit width combination corresponding to the neural network to be quantized is determined from the plurality of quantization bit width combinations.

[0010] According to another aspect of the present invention, a device for determining the quantization bit width of a neural network is provided, the device comprising:

[0011] A quantization sensitivity determination module is used to obtain the weight matrix of multiple network layers to be quantized in the neural network to be quantized, and to determine the quantization sensitivity of each network layer to be quantized based on the weight matrix of each network layer to be quantized, wherein the neural network to be quantized is a neural network trained based on a target task;

[0012] The layer group combination partitioning module is used to sort multiple network layers to be quantized according to the order of the quantization sensitivity, and to divide the sorted multiple network layers to be quantized into multiple layer group combinations according to the number of quantization bit width types, wherein each layer group combination includes multiple network layer groups.

[0013] A quantization bit width combination determination module is used to allocate the quantization bit width to each of the network layer groups in the multiple layer group combinations according to the quantization sensitivity of the network layer to be quantized in the layer group combination, so as to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized, wherein the quantization bit widths corresponding to the multiple network layers to be quantized in the network layer group are the same.

[0014] The target quantization bit width combination determination module is used to determine the target quantization bit width combination corresponding to the neural network to be quantized from multiple quantization bit width combinations based on the inference accuracy of the neural network to be quantized under each of the quantization bit width combinations.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the method for determining the quantization bit width of a neural network according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method for determining the quantization bit width of a neural network according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements a method for determining the quantization bit width of a neural network as described in any of the embodiments of this disclosure.

[0021] The technical solution of this invention firstly involves obtaining the weight matrices of multiple network layers in a neural network to be quantized, and determining the quantization sensitivity of each network layer based on its weight matrix. The neural network to be quantized is a neural network trained based on a target task. Training the neural network based on the target task lays a reliable foundation for quantization. Determining the quantization sensitivity through the weight matrix accurately reflects the degree of influence of the quantization process on different network layers, providing a scientific basis for subsequent differentiated quantization bit width allocation. Next, the multiple network layers to be quantized are sorted according to their quantization sensitivity, and then divided into multiple layer group combinations based on the number of quantization bit width types. Each layer group combination includes multiple network layer groups. Sorting by quantization sensitivity and combining it with the number of quantization bit width types to divide the layer group combinations yields multiple quantization bit width allocation schemes, enhancing the flexibility and adaptability of the quantization process. Finally, the quantization sensitivity of the network layers to be quantized in each layer group combination is determined based on their quantization sensitivity. The sensitivity is used to assign the quantization bit width to each of the network layer groups in a variety of layer group combinations to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized. The quantization bit widths corresponding to multiple network layers to be quantized within a network layer group are the same. Assigning quantization bit widths to network layer groups based on quantization sensitivity achieves a reasonable allocation of high quantization sensitivity layers with high quantization bit widths and low quantization sensitivity layers with low quantization bit widths. This ensures network performance while minimizing the accuracy loss caused by quantization, obtaining multiple quantization bit width combinations and providing sufficient options for subsequent selection of the optimal solution. Finally, the target quantization bit width combination corresponding to the neural network to be quantized is determined from the multiple quantization bit width combinations based on the inference accuracy of the neural network under each quantization bit width combination. Using inference accuracy as the core evaluation indicator, it is ensured that the determined target quantization bit width combination meets the inference accuracy requirements of the target task, guaranteeing the reliability and effectiveness of the final quantization scheme. This achieves the scientific and precise nature of the quantization process, effectively balancing the quantization accuracy and deployment efficiency of the neural network.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a method for determining the quantization bit width of a neural network according to Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a method for determining the quantization bit width of a neural network according to Embodiment 2 of the present invention;

[0026] Figure 3a This is a flowchart of a method for determining the quantization bit width of a neural network according to Embodiment 3 of the present invention;

[0027] Figure 3b This is a grouping diagram of a method for determining the quantization bit width of a neural network according to Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of a device for determining the quantization bit width of a neural network according to Embodiment 4 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the method for determining the quantization bit width of a neural network according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0034] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0035] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0036] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0037] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0038] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0039] Example 1

[0040] Figure 1 This is a flowchart illustrating a method for determining the quantization bit width of a neural network according to Embodiment 1 of the present invention. This embodiment is applicable to situations where the quantization bit width of a neural network needs to be determined. This method can be executed by a device for determining the quantization bit width of a neural network. This device can be implemented in hardware and / or software, optionally through an electronic device, such as a mobile terminal, a PC, or a server. Figure 1 As shown, the method may specifically include:

[0041] S110. Obtain the weight matrix of multiple network layers to be quantized in the neural network to be quantized, and determine the quantization sensitivity of each network layer to be quantized according to the weight matrix of each network layer to be quantized, wherein the neural network to be quantized is a neural network trained based on the target task.

[0042] In this embodiment of the invention, the quantization bit width of a neural network can be understood as the number of binary bits used when quantizing parameters such as weights and activation values ​​of the neural network. Under full-precision data format conditions, parameters such as weights and activation values ​​of a neural network are typically represented using 32-bit floating-point numbers. Quantization can convert these parameter values ​​into a lower bit-width representation, thereby reducing the size of the neural network by reducing the numerical precision in the neural network. The neural network to be quantized can be a neural network model trained based on a target task that requires quantization processing to determine its quantization bit width. The network layer to be quantized can be a network layer in the neural network to which quantization operations need to be performed. The weight matrix can be a parameter matrix stored in the network layer to be quantized, and its dimension is related to the input and output dimensions of the network layer to be quantized. The target task can be understood as the task that the neural network to be quantized is trained to complete, and the inference accuracy requirement of the target task is met through quantization operations.

[0043] Optionally, multiple layers of the neural network to be quantized can be determined, and the neural network to be quantized can be trained to determine the weight matrices of the multiple layers. Specifically, a large amount of sample data related to the target task can be obtained and its corresponding sample label results can be labeled. First, the weights of each layer of the neural network to be quantized are initialized. The training set data is input into the neural network to be quantized one by one. After propagation through the neural network to be quantized, multiple inference results are obtained. The multiple inference results are compared with the pre-labeled sample label results. The loss function corresponding to the neural network to be quantized is determined according to a preset algorithm. The loss function is backpropagated to the weights to update the weight matrices of each layer of the neural network to be quantized. Further, the weight gradient descent method can be used to train and update the weight matrices until the gradient is close to 0 and the loss converges to the loss threshold predetermined by the target task, completing the training process and obtaining the weight matrices of multiple networks of the neural network to be quantized that meet the expected accuracy threshold trained based on the target task.

[0044] Quantization sensitivity can be understood as the degree to which the inference accuracy of the network layer to be quantized is affected by changes in the quantization bit width after a quantization operation. Optionally, the change in the loss function when the weight matrix of the network layer to be quantized changes after a quantization operation can be calculated, characterizing the impact of quantization bit width error on inference accuracy, i.e., the quantization sensitivity of the network layer to be quantized. The neural network to be quantized is trained based on the target task, and its loss function has been reduced to a minimum through training. Quantization can measure the degree of damage to the weight matrix of the trained neural network to be quantized. The greater the quantization sensitivity, the greater the change in the loss function caused by changes in the weight matrix due to the quantization operation. Consequently, the network layer to be quantized requires a higher quantization bit width to reduce quantization error.

[0045] Based on the above scheme, optionally, determining the quantization sensitivity of each network layer to be quantized according to the weight matrix of each network layer to be quantized includes: for each network layer to be quantized, determining the Hessian matrix of the network layer to be quantized, determining multiple eigenvalues ​​of the Hessian matrix, and determining the quantization sensitivity of the network layer to be quantized according to the multiple eigenvalues.

[0046] The Hessian matrix represents a square matrix of the second-order partial derivatives of the loss function of the network layer to be quantized with respect to the weight parameters. It characterizes the curvature of the weight parameters and reflects the degree to which changes in the network layer parameters affect the loss function. Eigenvalues ​​are the numerical values ​​obtained after eigenvalue decomposition of the Hessian matrix, reflecting the scaling ratio of the linear transformation represented by the Hessian matrix.

[0047] In determining the quantization sensitivity, the quantization operation is performed on each network layer to be quantized, which can be viewed as the weight matrix of that network layer. Introducing a perturbation, namely the weight bias caused by the quantization operation, this perturbation affects the loss function of the neural network to be quantized. If a change occurs, the change in the loss function is denoted as... In order to approximate the depiction In the original weight matrix Expanding the loss function using a second-order Taylor series yields the following formula:

[0048] ;

[0049] in, This is the first-order partial derivative of the loss function with respect to the weight matrix, used to represent the weight gradient; This indicates the difference between the current value and the original value of the weight matrix after quantization. Deviation between; The Hessian matrix represents the weight matrix of the network layer to be quantized, which is the matrix formed by the second-order partial derivatives of the loss function with respect to the weight matrix. For a first-order term in a Taylor expansion; This is the second-order term of the Taylor expansion. Since the neural network to be quantized has already been pre-trained, the training process minimizes the loss function, therefore the original weights... Weight gradient at The first-order term in the Taylor expansion is close to 0. The change in the loss function is negligible; it is mainly due to the second-order term. The decision is then made by calculating the Hessian matrix of each network layer to be quantized. The Hessian matrix is ​​used to characterize the quantization sensitivity of each network layer to be quantized. The larger the second-order term corresponding to the Hessian matrix, the more significant the change in the loss function caused by quantization, that is, the more sensitive the network layer to be quantized is to quantization.

[0050] After determining the Hessian matrix, the eigenvalues ​​of the Hessian matrix can be calculated using various preset algorithms such as the power iteration method / inverse power iteration method, the random trace estimation method, and the low-rank approximation method.

[0051] Determining quantization sensitivity by using the Hessian matrix and its eigenvalues ​​can more accurately capture the influence of network layer parameters on the neural network output compared to simply relying on the numerical value of the weight matrix. The Hessian matrix can reflect the correlation between parameter changes and loss function changes, providing a more scientific basis for subsequent differentiated quantization that is closer to the essence of the neural network, and effectively avoiding the loss of quantization accuracy caused by inaccurate judgment of quantization sensitivity.

[0052] Based on the above scheme, optionally, determining the quantization sensitivity corresponding to the network layer to be quantized according to the plurality of feature values ​​includes: determining the maximum value of the plurality of feature values ​​as the quantization sensitivity corresponding to the network layer to be quantized; or, determining a plurality of target values ​​among the plurality of feature values, and determining the sum of the plurality of target values ​​as the quantization sensitivity corresponding to the network layer to be quantized, wherein the plurality of target values ​​are all greater than the feature values ​​other than the target values ​​among the plurality of feature values.

[0053] The target value is a subset of the eigenvalues ​​of the Hessian matrix whose values ​​are greater than the other eigenvalues. For example, it could be a subset of the largest eigenvalues.

[0054] Optionally, the maximum value of the calculated multiple feature values ​​can be used as the quantization sensitivity of the network layer to be quantized. Alternatively, the top K feature values ​​with the largest values ​​can be determined by the Top-K algorithm as target values, and the summation can be used to obtain the quantization sensitivity of the network layer to be quantized, thereby reducing computational resource overhead.

[0055] The two specific schemes for determining quantization sensitivity described above can balance computational efficiency and evaluation accuracy: using the maximum value of the feature value as the quantization sensitivity has simple calculation logic and low computational overhead, enabling efficient quantization; while selecting the sum of multiple larger target values ​​as the sensitivity can comprehensively consider multiple important parameters, avoid the one-sidedness that may exist in a single maximum value, improve the comprehensiveness and accuracy of quantization accuracy evaluation, and allow for flexible selection of the quantization sensitivity determination method according to actual needs, thereby improving the adaptability and practicality of quantization sensitivity calculation.

[0056] S120. Sort the multiple network layers to be quantized according to the order of their quantization sensitivity, and divide the sorted multiple network layers to be quantized into multiple layer group combinations according to the number of quantization bit width types, wherein each layer group combination includes multiple network layer groups.

[0057] The number of quantization bit width types refers to the number of different quantization bit widths that can be allocated to the network layer to be quantized. For example, when choosing 8-bit and 4-bit quantization bit widths, the number of types is 2. The number of quantization bit width types needs to be determined based on factors such as the hardware support capabilities and resource constraints of the target task deployment in the actual business scenario. A layer group combination can be a set of network layer groups obtained by dividing the sorted network layers to be quantized according to the number of quantization bit width types. Each layer group combination corresponds to a grouping method for the network layers to be quantized, providing different candidate schemes for subsequent bit width allocation. A network layer group can be understood as a set of network layers divided according to the grouping method corresponding to the layer group combination. Each network layer group corresponds to a unique quantization bit width. A network layer group may include at least one network layer to be quantized, or it may include zero network layers to be quantized.

[0058] Optionally, multiple network layers to be quantized can be sorted sequentially according to the determined order of quantization sensitivity. Based on the number of optional quantization bit widths provided by the hardware corresponding to the target task, the sorted multiple network layers to be quantized can be divided into multiple network layer groups based on multiple preset partitioning methods. Each preset partitioning method corresponds to a layer group combination, that is, the allocation scheme of network layers to be quantized and quantization bit widths. The layer group combination includes multiple network layer groups.

[0059] Optionally, the number of network layer groups can be the same as the number of quantization bit widths. For example, multiple network layers to be quantized, sorted by quantization sensitivity from largest to smallest, can be considered as a corresponding number of... The first pattern symbol is determined, and the number of types is the same as the quantization bit width. The second pattern symbol will The second pattern symbol is inserted as a group separator into the ordered arrangement. Divide all the first pattern symbols into groups within the intervals of the first pattern symbols or on both sides. A network layer group, wherein the network layer group may include 0 first pattern symbols, but all first pattern symbols must be assigned to a certain network layer group; The first pattern symbol and The second pattern symbol together constitutes One location is available. The second pattern symbol is placed, and there are a total of [number] existing [symbols]. There are several placement schemes, each corresponding to a layer group combination, i.e., there exist... Layer group combination.

[0060] Optionally, the number of network layer groups can be less than the number of quantization bit width types. For example, the number of quantization bit width types can be determined based on the number of quantization bit width types. Determine the number of network layer groups ,in The sorted network layers to be quantized are divided into several pre-defined partitioning methods, such as equal-interval partitioning and segmentation based on sensitivity range thresholds. A network layer group, which may include at least one network layer to be quantized, and all network layers to be quantized within the same network layer group will be assigned the same quantization bit width. Select from the optional quantization bit widths Each quantization bit width, respectively with Each network layer group establishes a one-to-one correspondence, generating a layer group combination under the preset partitioning method.

[0061] S130. Assign the quantization bit width to each of the multiple layer group combinations according to the quantization sensitivity of the network layer to be quantized in the layer group combination, so as to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized, wherein the quantization bit widths corresponding to multiple network layers to be quantized in the network group are the same.

[0062] Among them, the quantization bit width combination refers to the configuration scheme of the quantization bit width of each layer of multiple neural networks to be quantized after allocating the quantization bit width to each network layer group in the layer group combination.

[0063] Optionally, for each network layer group in multiple layer group combinations, a quantization bit width can be allocated to each network layer group according to the quantization sensitivity of the network layer to be quantized in the layer group combination. The allocation of the quantization bit width to multiple network layer groups satisfies a preset bit width allocation condition. The preset bit width allocation condition includes that the quantization bit width allocated to the network layer to be quantized with higher quantization sensitivity is not lower than the quantization bit width allocated to the network layer to be quantized with lower quantization sensitivity, thereby obtaining multiple quantization bit width combinations corresponding to the neural network to be quantized and realizing on-demand allocation.

[0064] In this context, the quantization bit widths of the multiple network layers to be quantized within the network layer group are the same. If the network layer group does not include a network layer to be quantized, it means that its corresponding quantization bit width has not been allocated.

[0065] By setting scientific constraints on quantization bit width allocation through preset bit width allocation conditions, it ensures that more sufficient quantization accuracy support is provided for performance-critical network layers, while allowing appropriate reduction of bit width for network layers with low quantization sensitivity to save resources. This avoids performance waste or accuracy loss caused by misallocation and provides a clear standard for bit width allocation, making quantization bit width allocation logical and reasonable.

[0066] S140. Determine the target quantization bit width combination corresponding to the neural network to be quantized from the plurality of quantization bit width combinations based on the inference accuracy of the neural network to be quantized under each of the quantization bit width combinations.

[0067] Inference accuracy can be understood as the proportion of correct results output by the neural network to be quantized when performing the target task after quantization using a certain combination of quantization bit widths. The target quantization bit width combination can be a quantization bit width configuration scheme that meets preset requirements and is selected from multiple quantization bit width combinations.

[0068] Optionally, the inference accuracy of the neural network to be quantized can be calculated for each quantization bit width combination. Based on the expected accuracy threshold of the target task, the target quantization bit width combination corresponding to the neural network to be quantized can be determined from multiple quantization bit width combinations. This ensures that the neural network model corresponding to the target quantization bit width combination can complete the target task normally and avoids the model performance from failing to meet the target due to excessive pursuit of resource saving. At the same time, if there are multiple quantization bit width combinations that meet the expected accuracy threshold, the optimal solution can be selected by combining indicators such as storage overhead and computational efficiency to determine the target quantization bit width combination, thereby achieving a balance between model performance and computational resources.

[0069] The technical solution of this invention firstly involves obtaining the weight matrices of multiple network layers in a neural network to be quantized, and determining the quantization sensitivity of each network layer based on its weight matrix. The neural network to be quantized is a neural network trained based on a target task. Training the neural network based on the target task lays a reliable foundation for quantization. Determining the quantization sensitivity through the weight matrix accurately reflects the degree of influence of the quantization process on different network layers, providing a scientific basis for subsequent differentiated quantization bit width allocation. Next, the multiple network layers to be quantized are sorted according to their quantization sensitivity, and then divided into multiple layer group combinations based on the number of quantization bit width types. Each layer group combination includes multiple network layer groups. Sorting by quantization sensitivity and combining it with the number of quantization bit width types to divide the layer group combinations yields multiple quantization bit width allocation schemes, enhancing the flexibility and adaptability of the quantization process. Finally, the quantization sensitivity of the network layers to be quantized in each layer group combination is determined based on their quantization sensitivity. The sensitivity is used to assign the quantization bit width to each of the network layer groups in a variety of layer group combinations to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized. The quantization bit widths corresponding to multiple network layers to be quantized within a network layer group are the same. Assigning quantization bit widths to network layer groups based on quantization sensitivity achieves a reasonable allocation of high quantization sensitivity layers with high quantization bit widths and low quantization sensitivity layers with low quantization bit widths. This ensures network performance while minimizing the accuracy loss caused by quantization, obtaining multiple quantization bit width combinations and providing sufficient options for subsequent selection of the optimal solution. Finally, the target quantization bit width combination corresponding to the neural network to be quantized is determined from the multiple quantization bit width combinations based on the inference accuracy of the neural network under each quantization bit width combination. Using inference accuracy as the core evaluation indicator, it is ensured that the determined target quantization bit width combination meets the inference accuracy requirements of the target task, guaranteeing the reliability and effectiveness of the final quantization scheme. This achieves the scientific and precise nature of the quantization process, effectively balancing the quantization accuracy and deployment efficiency of the neural network.

[0070] Example 2

[0071] Figure 2 This is a flowchart illustrating a method for determining the quantization bit width of a neural network according to Embodiment 2 of the present invention. It further describes a specific implementation method for determining the target quantization bit width combination corresponding to the neural network to be quantized from multiple quantization bit width combinations based on the inference accuracy of the neural network under each quantization bit width combination. Specific implementation methods can be found in the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here. Figure 2 As shown, the method may specifically include:

[0072] S210. Obtain the weight matrix of multiple network layers to be quantized in the neural network to be quantized, and determine the quantization sensitivity of each network layer to be quantized according to the weight matrix of each network layer to be quantized, wherein the neural network to be quantized is a neural network trained based on the target task.

[0073] S220. Sort the multiple network layers to be quantized according to the order of their quantization sensitivity, and divide the sorted multiple network layers to be quantized into multiple layer group combinations according to the number of quantization bit width types, wherein each layer group combination includes multiple network layer groups.

[0074] S230. Assign the quantization bit width to each of the multiple layer group combinations according to the quantization sensitivity of the network layer to be quantized in the layer group combination, so as to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized, wherein the quantization bit widths corresponding to multiple network layers to be quantized in the network group are the same.

[0075] S240. Determine the storage overhead of each of the quantization bit width combinations, and based on the storage overhead of the quantization bit width combination and the inference accuracy of the neural network to be quantized under the quantization bit width combination, determine the target quantization bit width combination corresponding to the neural network to be quantized from the multiple quantization bit width combinations.

[0076] Storage overhead can be understood as the amount of storage space required to store all the weight matrix parameters of the network layers after the neural network to be quantized is quantized using a certain combination of quantization bit widths.

[0077] Optionally, a target quantization bit width combination corresponding to the neural network to be quantized can be selected from multiple quantization bit width combinations based on the storage overhead of each quantization bit width combination and the inference accuracy of the neural network to be quantized under that quantization bit width combination.

[0078] Based on the above scheme, optionally, determining the storage overhead of each of the quantization bit width combinations includes: for each of the quantization bit width combinations, determining the storage overhead of the quantization bit width combination based on the quantization bit width corresponding to the multiple network layers to be quantized in the quantization bit width combination and the number of weights in the network layers to be quantized.

[0079] The number of weights refers to the number of parameters contained in the weight matrix of the network layer to be quantized, which can be obtained by multiplying the number of rows and columns of the weight matrix.

[0080] The storage overhead calculation based on quantization bit width and number of weights is highly operable and accurate. It directly relates to the core quantization bit width parameter of the network layer and the basic weight bit width occupied by the hardware storage, avoiding evaluation bias caused by ambiguity in the storage overhead calculation logic. At the same time, it calculates and sums the data for each network layer to be quantized, accurately locating the contribution of different network layers to the overall storage overhead, providing clear data support for subsequent selection of quantization bit width combinations based on storage overhead.

[0081] Specifically, for each quantization bit width combination, determining the storage overhead of the quantization bit width combination based on the quantization bit width corresponding to the plurality of network layers to be quantized in the quantization bit width combination and the number of weights in the network layers to be quantized includes: determining the quantization bit width corresponding to each network layer to be quantized in the quantization bit width combination and the number of weights in the network layers to be quantized; for each network layer to be quantized, determining the product of the quantization bit width corresponding to the network layer to be quantized and the number of weights as the storage overhead of the network layer to be quantized under the quantization bit width combination; and determining the sum of the storage overheads of the plurality of network layers to be quantized as the storage overhead of the quantization bit width combination.

[0082] Optionally, the number of parameters of the network layer to be quantized can be obtained as the number of its corresponding weights, and the quantization bit width corresponding to each network layer to be quantized under the combination of quantization bit widths can be determined.

[0083] Furthermore, the storage overhead for each combination of quantization bit widths can be calculated based on the following formula:

[0084] ;

[0085] in, This indicates the storage overhead under the combination of quantization bit widths; This indicates the number of network layers to be quantized; Indicates the first The number of weights in each network layer to be quantized; Indicates the first The quantization bit width allocated to each network layer to be quantized under this quantization bit width combination.

[0086] Optionally, a target quantization bit width combination corresponding to the neural network to be quantized can be determined based on the inference accuracy and storage overhead of the neural networks to be verified corresponding to multiple quantization bit width combinations. Specifically, the inference accuracy and storage overhead of the neural networks to be verified corresponding to multiple quantization bit width combinations can be normalized to convert them into values ​​under the same dimension. The weights of inference accuracy and storage overhead are determined according to the priority requirements of the target task for performance and computing resources. The quantization bit width combination index is calculated by weighted summation, and the target quantization bit width combination is selected based on the obtained multiple quantization bit width combination indices.

[0087] The above storage overhead calculation achieves a refined and transparent assessment of storage overhead, intuitively reflecting the storage requirements of each network layer. The total storage overhead is obtained by summing the overhead of all layers, making the comparison of storage overhead for different quantization bit width combinations more intuitive and providing accurate and comparable quantitative data for subsequent screening.

[0088] Based on the above scheme, optionally, determining the target quantization bit width combination corresponding to the neural network to be quantized from multiple quantization bit width combinations according to the storage overhead of the quantization bit width combination and the inference accuracy of the neural network to be quantized under the quantization bit width combination includes: determining the quantization bit width combination to be verified from multiple quantization bit width combinations according to the storage overhead of the multiple quantization bit width combinations; quantizing multiple network layers to be quantized in the neural network to be quantized based on the quantization bit width combination to be verified to obtain the neural network to be verified; performing multiple inference operations associated with the target task based on the neural network to be verified to obtain multiple network inference results; determining the inference accuracy of the neural network to be verified based on the multiple network inference results; and determining the target quantization bit width combination corresponding to the neural network to be quantized based on the inference accuracy of the neural network to be verified.

[0089] Here, the quantization bit width combination to be verified can be understood as a quantization bit width configuration scheme selected from multiple quantization bit width combinations, whose performance needs to be verified through inference execution on an actual target task. The neural network to be verified can be a neural network model obtained by quantizing the original neural network to be quantized using the quantization bit width combination to be verified. The network inference result refers to the output result of the neural network to be verified after performing inference operations.

[0090] Specifically, determining the quantization bit width combination to be verified among the multiple quantization bit width combinations based on the storage overhead of the multiple quantization bit width combinations includes: sorting the multiple quantization bit width combinations according to the size of their storage overhead, and sequentially obtaining the quantization bit width combination to be verified from the sorted multiple quantization bit width combinations in ascending order of storage overhead; quantizing multiple network layers in the neural network to be quantized based on the quantization bit width combination to be verified to obtain the neural network to be verified; performing multiple inference operations associated with the target task based on the neural network to be verified to obtain multiple network inference results, and determining the inference accuracy of the neural network to be verified based on the multiple network inference results; further, determining the target quantization bit width combination corresponding to the neural network to be quantized based on the inference accuracy of the neural network to be verified includes: in response to an event that the inference accuracy reaches an expected accuracy threshold, determining the quantization bit width combination to be verified as the target quantization bit width combination corresponding to the neural network to be quantized.

[0091] By filtering quantization bit width combinations to be verified from smallest to largest storage overhead, the optimal solution with low storage overhead and high inference accuracy can be quickly located, avoiding wasting computational resources on combinations with high storage overhead. At the same time, using the expected accuracy threshold as the termination criterion makes the determination of the target combination more objective, without having to traverse all combinations. Once a combination that meets the accuracy requirement and has the minimum storage overhead is found, the process can be terminated. This ensures the practicality of the quantization scheme, optimizes storage resources to the greatest extent, and improves the efficiency of the quantization process.

[0092] Optionally, multiple quantization bit width combinations that meet the storage overhead threshold can be selected based on the storage overhead of multiple quantization bit width combinations and a preset storage overhead threshold. Then, the quantization bit width combination to be verified is used to quantize the quantized neural network to obtain the neural network to be verified and perform multiple inference operations associated with the target task to obtain multiple network inference results. The inference accuracy of the neural network to be verified is determined based on the multiple network inference results. When the inference accuracy reaches the expected accuracy threshold for the first time, the search for the quantization bit width combination to be verified is stopped and it is determined as the target quantization bit width combination. Alternatively, after calculating the inference accuracy of the neural network to be verified corresponding to multiple quantization bit width combinations to be verified, the neural network to be verified corresponding to the inference accuracy is determined as the target quantization bit width combination.

[0093] By screening the quantization bit width combinations to be verified, blind verification of all combinations is avoided, significantly saving computational resources. Multiple inference operations are performed based on the target task, effectively avoiding the randomness of single inference results. The inference accuracy is determined through statistical analysis of multiple results, ensuring that the evaluation results are more in line with the performance of the model in the actual application scenario. This avoids the selection of an inappropriate quantization scheme due to single verification errors, and greatly improves the reliability of the target quantization bit width combination.

[0094] The technical solution of this invention determines the storage overhead of each quantization bit width combination, and determines the target quantization bit width combination corresponding to the neural network to be quantized from multiple quantization bit width combinations based on the storage overhead of the quantization bit width combination and the inference accuracy of the neural network to be quantized under the quantization bit width combination. By using storage overhead and inference accuracy as evaluation indicators, storage overhead can be reduced while ensuring inference accuracy, achieving a dual balance between inference accuracy and storage overhead.

[0095] Example 3

[0096] As an optional example of an embodiment of the present invention, taking the star-bar method as an example, the method for determining the quantization bit width of the neural network of the present invention may specifically include:

[0097] Step 1: Train the target neural network under full-precision data format conditions to obtain the weight values ​​of each layer that meet the inference accuracy requirements under the target task.

[0098] Under full-precision data format, the weights, activation values, and other parameters of a neural network are typically represented using 32-bit floating-point numbers, i.e., 32-bit single-precision floating-point numbers (FP32). A sample training set is obtained to train the target neural network. After initializing the weights of each layer, the sample training set data is input into the neural network. After propagation through the neural network, the inference result is obtained. The inference result is compared with the pre-labeled result of the sample training set. For example, for an image classification task, the sample training set may include image data and corresponding classification labels. After classifying the image data, the neural network compares it with the preset classification labels to determine the loss between the inference result of the entire sample training set and the pre-labeled result. This loss is backpropagated to the weights, and the weight values ​​are updated. Gradient descent can be used to train and update the weights until the gradient approaches 0, the loss converges to an acceptable level, and the training is completed. Finally, the weight values ​​of each layer that meet the inference accuracy requirements for the target task are obtained.

[0099] Step 2: Calculate the quantization sensitivity of each layer. This is done by calculating the degree of change in the loss function when the weights change slightly, which characterizes the impact of quantization operation error on inference accuracy. This characterizes the importance of quantization precision (i.e., quantization bit width) to the weights of that layer. Based on the magnitude of the quantization sensitivity, each network layer to be quantized is sorted from largest to smallest to form a sequence. The number of elements in the sequence is the number of network layers to be quantized.

[0100] The quantization sensitivity of each layer is calculated, which characterizes the impact of quantization error on inference accuracy by calculating the degree of change in the loss function when the weights undergo small changes. For example, for image classification tasks, the loss function can be specifically represented as the cross-entropy loss function, etc., to measure the difference between the probability distribution predicted by the neural network model and the true probability distribution. Since the loss function has been reduced to a minimum value through training in step 1, quantization can measure the degree of damage to the trained model weights. Therefore, the greater the quantization sensitivity, the greater the change in the loss function caused by changes in the weights due to quantization. Thus, the weights of that layer need higher quantization precision (i.e., quantization bit width) to reduce quantization error.

[0101] The network layers to be quantized are sorted from largest to smallest according to their quantization sensitivity, forming a sequence. The number of elements in the sequence represents the number of network layers to be quantized. Taking an 18-layer residual neural network (Residual Network-18, ResNet-18) as an example, this network model includes the following layers in the forward propagation order of the data flow: the initial convolutional layer conv1, the four convolutional layers layer1.conv1 and layer1.conv4 of residual block 1, the first two convolutional layers layer2.conv1 and layer2.conv2 of residual block 2, the downsampling layer layer2.downsample and the normalization layer layer2.downsample.1 of residual block 2, the last two convolutional layers layer2.conv3 and layer2.conv4 of residual block 2, and the first two convolutional layers layer3.conv1 and layer2.conv4 of residual block 3. 3. The network consists of 24 layers to be quantized, including the convolutional layer layer3.downsample.0 and the normalization layer layer3.downsample.1 of the downsampling layer of residual block 2, the last two convolutional layers layer3.conv3 and layer3.conv4 of residual block 2, the first two convolutional layers layer4.conv1 and layer4.conv2 of residual block 4, the downsampling layer layer4.downsample.0 and the normalization layer layer4.downsample.1 of residual block 4, the last two convolutional layers layer4.conv3 and layer4.conv4 of residual block 4, and the fully connected layer fc.

[0102] Alternatively, the Hessian matrix method can be used to calculate the quantization sensitivity. The weights of a certain layer, after quantization, can be viewed as the weights in the weight matrix of that layer. Introduce a perturbation that affects the loss function of the neural network. If a change occurs, the change in the loss function is denoted as... In the weight matrix Approximate the result by performing a second-order Taylor series expansion at the given location. ,in It is the weight gradient. For quantization error, It is the Hessian matrix, which is the second derivative of the weight matrix. It is a first-order term in a Taylor expansion. This is the second-order term in the Taylor expansion. Since the neural network has already been trained in step 1, the weight gradient is close to 0 at this point, so the change in the loss function is mainly determined by the second-order term. Therefore, the quantization sensitivity of each layer's weights can be characterized by calculating the Hessian matrix of each layer's weights. Further, we first calculate the eigenvalues ​​of the Hessian matrix, and then calculate the sum of the K largest eigenvalues, denoted as Top-K, as the quantization sensitivity to reduce computational resource overhead. For example, the quantization sensitivity can be calculated using the largest eigenvalue or the sum of the top 10 eigenvalues, i.e., the Top-1 or Top-10 method.

[0103] The Top-1 feature values ​​of each of the 24 layers of the ResNet-18 neural network were calculated using the method described above as the quantization sensitivity of each layer. These 24 layers were then reordered according to their quantization sensitivity from largest to smallest, forming a sequence of 24 elements. Table 1 shows the sorting of the quantization sensitivities calculated for each layer's weights under the condition of training the ResNet-18 network for 100 epochs using the Canadian Institute for Advanced Research 10 (Cifa-10) dataset. It can be seen that this sorting differs from the order of forward propagation of the data flow.

[0104] Serial Number Layer name Parameters Quantitative sensitivity Serial Number Layer name Parameters Quantitative sensitivity 1 conv1 5,120 1 13 layer4.downsample.0 131,072 0.0578 2 fc 2,359,296 0.4428 14 layer3.downsample.0 32,768 0.0504 3 layer1.conv1 36,864 0.2774 15 layer4.conv3 2,359,296 0.0479 4 layer4.conv4 2,359,296 0.2665 16 layer2.conv4 147,456 0.0444 5 layer4.conv1 1,179,648 0.2289 17 layer3.conv4 589,824 0.0344 6 layer1.conv2 36,864 0.2088 18 layer2.conv1 73,728 0.0252 7 layer3.conv3 589,824 0.1896 19 layer2.conv2 147,456 0.0217 8 layer3.conv2 589,824 0.1589 20 layer4.downsample.1 512 0.0110 9 layer1.conv3 36,864 0.1273 21 layer2.downsample.1 128 0.0012 10 layer4.conv2 2,359,296 0.1222 22 layer3.downsample.1 256 0.0008 11 layer1.conv4 36,864 0.0936 23 layer2.conv3 147,456 -0.0031 12 layer3.conv1 294,912 0.0579 24 layer2.downsample.0 8,192 -0.0132

[0105] Table 1 Ranking of Sensitivity to Weight Quantization at Each Layer

[0106] Step 3, based on the optional quantization bit width types provided by the hardware. The elements in the sequence from step 2 are then processed using the star-bar method. Without changing the sorting, high bit widths are assigned to layers with high quantization sensitivity and low bit widths are assigned to layers with low quantization sensitivity, resulting in a variety of quantization bit width combinations and forming a search space for quantization bit width combinations.

[0107] Specifically, the following steps are included:

[0108] Step 3.1: Sort the data by quantization sensitivity from highest to lowest. Each element is considered as The symbol "*" (star).

[0109] Step 3.2, will Each "|" is inserted sequentially into the ordered sequence. Among the asterisks, divide all the asterisks into Groups; a group can contain 0 "*" elements, which means that the corresponding quantization bit width has not been allocated, but all "*" elements must be assigned to a group; The "*" element and The "|" elements together constitute One location is available. Place each "|" symbol to form a total of There are several placement methods.

[0110] Step 3.3, divide into Each group is allocated the available hardware resources sequentially from high-bit width to low-bit width. With a certain quantization bit width, the quantization can be completed. Layers sorted by quantization sensitivity from largest to smallest are grouped according to quantization bit width from highest to lowest.

[0111] Taking four quantization bit widths as examples—32-bit single-precision floating-point (FP32), 8-bit integer (INT8), 4-bit integer (INT4), and 2-bit integer (INT2)—the number of types... For the ResNet-18 neural network, the 24 network layers to be quantized, sorted from largest to smallest quantization sensitivity, are divided into 4 groups.

[0112] The asterisk method is used to divide a sequence of 24 elements into 4 groups. The specific steps are as follows: Treat the 24 elements, sorted by sensitivity from highest to lowest, as 24 asterisks (*); insert 3 asterisks (|) into the intervals between the 24 ordered asterisks or on either side, dividing all asterisks into 4 groups; a group can contain 0 asterisks, representing a corresponding quantization bit width that has not been allocated, but all asterisks must be assigned to a group; the 24 asterisks and 3 asterisks together form 27 positions, which can be used to place the 3 asterisks. There are several placement methods. Finally, the four groups are assigned the four quantization bit widths available in the hardware in order from high bit width to low bit width, thus completing the grouping of the 24 layers sorted from highest to lowest sensitivity according to the quantization bit width from highest to lowest.

[0113] Step 4: Calculate the storage overhead of all quantization bit width combinations in Step 3, and sort them in ascending order;

[0114] Optionally, the storage overhead mentioned in step 4 is equal to the sum of the weights of each layer multiplied by their corresponding quantization bit widths, which can be expressed as follows: ,in Indicates the first The number of weights in each network layer to be quantized; Indicates the first The quantization bit width allocated to each network layer to be quantized under this quantization bit width combination.

[0115] Continuing with the ResNet-18 neural network example, each weight in FP32 data format occupies 4 bytes, INT16 2 bytes, INT8 1 byte, and INT2 1 / 4 byte. If the quantization width allocated to the conv1 layer is FP32, according to Table 1, the number of parameters in the conv1 layer is 5120, and the number of bytes occupied by FP32 is 4. Therefore, the storage space occupied by this layer is 5120 × 4 = 20480 bytes = 20 KB. If the quantization width allocated to the conv1 layer is INT2, then the storage space occupied by this layer is 5120 × 1 / 4 = 1280 bytes = 1.25 KB. Similarly, based on the actual allocated quantization width and the number of weight parameters, the storage overhead of each layer can be calculated, thus obtaining the total storage overhead for each combination of quantization widths.

[0116] Step 5: Following the sorting in Step 4, begin the quantization bit width search, that is, quantize the deep neural network sequentially according to each quantization bit width combination, and calculate the inference accuracy of the quantized neural network; when the inference accuracy reaches the expected accuracy threshold, record the corresponding quantization bit width combination and stop the bit width search; the quantization bit width combination recorded at this time is the quantization bit width combination with the minimum storage overhead under the condition of meeting the accuracy requirements, and the search is completed.

[0117] Taking the ResNet-18 neural network as an example, it was calculated that the total storage overhead is minimized when all 24 network layers to be quantized are assigned an INT2 quantization bit width, and maximized when all 24 network layers to be quantized are assigned an FP32 quantization bit width.

[0118] When the inference accuracy reaches the expected accuracy threshold (which can be set according to the target task requirements in the actual business scenario), the corresponding quantization bit width combination is recorded, and the bit width search is stopped. The quantization bit width combination recorded at this time is the quantization bit width combination with the minimum storage overhead under the condition of meeting the accuracy requirements, and the search is completed.

[0119] Step 6: Quantize the neural network weights according to the searched quantization bit width combination and deploy them to the hardware corresponding to the target task.

[0120] The technical solution of this invention first trains the target neural network under full-precision data format conditions to obtain the weight values ​​of each layer that meet the inference accuracy requirements of the target task. The neural network training provides a high-precision weight foundation for subsequent quantization operations. Next, the quantization sensitivity of each layer is calculated, and the layers to be quantized are sorted from largest to smallest according to the magnitude of the quantization sensitivity. Based on the optional quantization bit width types provided by the hardware, high bit widths are assigned to layers with high quantization sensitivity, and low bit widths are assigned to layers with low quantization sensitivity, resulting in multiple quantization bit width combinations. This forms a search space for quantization bit width combinations. Based on the quantization sensitivity calculation, the impact of quantization operations on each network layer is accurately identified. The search space is constructed according to the quantization bit width types, providing sufficient quantization combination schemes for subsequent optimization. Next, the storage overhead of all quantization bit width combinations is calculated, and they are sorted from smallest to largest. The inference accuracy of the neural network after quantization of each quantization bit width combination is calculated based on the sorting. The final quantization bit width combination is determined according to the expected accuracy threshold. By selecting quantization bit width combinations based on storage overhead and expected accuracy, a balance is achieved between model lightweighting and inference accuracy, providing accuracy and practicality in quantization bit width determination.

[0121] Example 4

[0122] Figure 4 This is a schematic diagram of a device for determining the quantization bit width of a neural network according to Embodiment 4 of the present invention. This device is used to execute the method for determining the quantization bit width of a neural network provided in any of the above embodiments. This device and the method for determining the quantization bit width of a neural network in the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the device for determining the quantization bit width of a neural network can be referred to the embodiments of the method for determining the quantization bit width of a neural network. Figure 4 As shown, the device includes: a quantization sensitivity determination module 410, a layer group combination division module 420, a quantization bit width combination determination module 430, and a target quantization bit width combination determination module 440.

[0123] The quantization sensitivity determination module 410 is used to obtain the weight matrices of multiple network layers to be quantized in the neural network to be quantized, and to determine the quantization sensitivity of each network layer to be quantized based on the weight matrix of each network layer to be quantized, wherein the neural network to be quantized is a neural network trained based on a target task; the layer group combination partitioning module 420 is used to sort the multiple network layers to be quantized according to the order of the quantization sensitivity, and to divide the sorted multiple network layers to be quantized into multiple layer group combinations according to the number of quantization bit width types, wherein each layer group combination includes multiple network layers. The module 430 is configured to allocate the quantization bit width to each of the network layer groups in the multiple layer group combinations according to the quantization sensitivity of the network layer to be quantized in the layer group combination, so as to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized, wherein the quantization bit widths corresponding to multiple network layers to be quantized in the network layer group are the same; the module 440 is configured to determine the target quantization bit width combination corresponding to the neural network to be quantized from the multiple quantization bit width combinations according to the inference accuracy of the neural network to be quantized under each quantization bit width combination.

[0124] The technical solution of this invention embodiment firstly involves obtaining the weight matrices of multiple network layers to be quantized in the neural network to be quantized through the quantization sensitivity determination module 410, and determining the quantization sensitivity of each network layer to be quantized based on the weight matrix of each network layer to be quantized. The neural network to be quantized is a neural network trained based on a target task. Training the neural network based on the target task lays a reliable foundation for quantization work. Determining the quantization sensitivity through the weight matrix accurately reflects the degree of influence of the quantization process on different network layers, providing a scientific basis for subsequent differentiated quantization bit width allocation. Next, the layer group combination partitioning module 420 sorts the multiple network layers to be quantized according to the order of the quantization sensitivity, and divides the sorted multiple network layers to be quantized into multiple layer group combinations according to the number of quantization bit width types. Each layer group combination includes multiple network layer groups. Sorting by quantization sensitivity and dividing the layer group combinations according to the number of quantization bit width types yields multiple quantization bit width allocation schemes, enhancing the flexibility and adaptability of the quantization process. Finally, the quantization bit width combination determination module 430 determines the quantization sensitivity of each network layer according to the weight matrix of the layer group combination. The quantization sensitivity of the quantization network layers is used to assign the quantization bit width to each of the network layer groups in a variety of layer group combinations, thereby obtaining multiple quantization bit width combinations corresponding to the neural network to be quantized. The quantization bit widths corresponding to multiple network layers to be quantized within a network layer group are the same. Assigning quantization bit widths to network layer groups based on quantization sensitivity achieves a reasonable allocation of high quantization sensitivity layers with high quantization bit widths and low quantization sensitivity layers with low quantization bit widths. This ensures network performance while minimizing the accuracy loss caused by quantization, and provides ample options for subsequent selection of the optimal solution. Finally, the target quantization bit width combination determination module 440 determines the target quantization bit width combination corresponding to the neural network to be quantized from the multiple quantization bit width combinations based on the inference accuracy of the neural network under each quantization bit width combination. Using inference accuracy as the core evaluation indicator, it ensures that the determined target quantization bit width combination meets the inference accuracy requirements of the target task, guaranteeing the reliability and effectiveness of the final quantization scheme. This achieves the scientific and precise nature of the quantization process, effectively balancing the quantization accuracy and deployment efficiency of the neural network.

[0125] Based on the above scheme, optionally, the quantization sensitivity determination module 410 includes a quantization sensitivity determination submodule. The quantization sensitivity determination submodule is used to determine the Hessian matrix of each network layer to be quantized, determine multiple eigenvalues ​​of the Hessian matrix, and determine the quantization sensitivity corresponding to the network layer to be quantized based on the multiple eigenvalues.

[0126] Based on the above scheme, optionally, the quantization sensitivity determination submodule includes a first quantization sensitivity determination unit and a second quantization sensitivity determination unit. The first quantization sensitivity determination unit is used to determine the maximum value of a plurality of the aforementioned feature values ​​as the quantization sensitivity corresponding to the network layer to be quantized; the second quantization sensitivity determination unit is used to determine a plurality of target values ​​among the plurality of feature values, and to determine the sum of the plurality of target values ​​as the quantization sensitivity corresponding to the network layer to be quantized, wherein all of the plurality of target values ​​are greater than the feature values ​​other than the target values ​​among the plurality of feature values.

[0127] Optionally, based on the above scheme, the target quantization bit width combination determination module 440 includes a target quantization bit width combination determination submodule. This submodule is used to determine the storage overhead of each of the quantization bit width combinations, and based on the storage overhead of the quantization bit width combination and the inference accuracy of the neural network to be quantized under that quantization bit width combination, determine the target quantization bit width combination corresponding to the neural network to be quantized from among the multiple quantization bit width combinations.

[0128] Optionally, based on the above scheme, the target quantization bit width combination determination submodule includes a storage overhead determination unit. The storage overhead determination unit is used to determine the storage overhead of each quantization bit width combination based on the quantization bit width corresponding to the multiple network layers to be quantized in the quantization bit width combination and the number of weights in the network layers to be quantized.

[0129] Based on the above scheme, optionally, the storage overhead determination unit includes a network layer information determination subunit, a single storage overhead determination subunit, and a combined storage overhead determination subunit. The network layer information determination subunit is used to determine the quantization bit width and the number of weights in each network layer to be quantized in the quantization bit width combination; the single storage overhead determination subunit is used to determine the storage overhead of each network layer to be quantized by multiplying the quantization bit width and the number of weights in that network layer by that product; the combined storage overhead determination subunit is used to determine the storage overhead of the quantization bit width combination by summing the storage overhead of multiple network layers to be quantized.

[0130] Based on the above scheme, optionally, the target quantization bit width combination determination submodule includes a neural network to be verified unit and a target quantization bit width combination determination unit. The neural network to be verified unit is used to determine the quantization bit width combination to be verified from the plurality of quantization bit width combinations based on the storage overhead of the plurality of quantization bit width combinations, and to quantize the plurality of network layers to be quantized in the neural network to be quantized based on the quantization bit width combination to obtain the neural network to be verified. The target quantization bit width combination determination unit is used to perform multiple inference operations associated with the target task based on the neural network to be verified to obtain multiple network inference results, to determine the inference accuracy of the neural network to be verified based on the multiple network inference results, and to determine the target quantization bit width combination corresponding to the neural network to be quantized based on the inference accuracy of the neural network to be verified.

[0131] Based on the above scheme, optionally, the neural network to be verified determination unit includes a quantization bit width combination to be verified determination subunit; wherein, the quantization bit width combination to be verified determination subunit is used to sort multiple quantization bit width combinations according to the size of their storage overhead, and sequentially obtain the quantization bit width combination to be verified from the sorted multiple quantization bit width combinations in order of ascending storage overhead.

[0132] Optionally, based on the above scheme, the target quantization bit width combination determination unit includes a target quantization bit width combination determination subunit. The target quantization bit width combination determination subunit is used to determine the quantization bit width combination to be verified as the target quantization bit width combination corresponding to the neural network to be quantized, in response to an event that the inference accuracy reaches the expected accuracy threshold.

[0133] Based on the above scheme, optionally, the quantization bit widths of the multiple network layer groups are allocated to meet preset bit width allocation conditions, wherein the preset bit width allocation conditions include that the quantization bit width allocated to the network layer to be quantized with higher quantization sensitivity is not lower than the quantization bit width allocated to the network layer to be quantized with lower quantization sensitivity.

[0134] The neural network quantization bit width determination device provided in the embodiments of the present invention can execute the neural network quantization bit width determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0135] Example 5

[0136] Figure 5A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0137] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0138] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for determining the quantization bit width of a neural network.

[0140] In some embodiments, the method for determining the quantization bit width of a neural network can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for determining the quantization bit width of a neural network described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for determining the quantization bit width of a neural network by any other suitable means (e.g., by means of firmware).

[0141] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0145] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0146] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0147] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0148] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining the quantization bit width of a neural network, characterized in that, include: Obtain the weight matrices of multiple network layers to be quantized in the neural network to be quantized, and determine the quantization sensitivity of each network layer to be quantized based on the weight matrix of each network layer to be quantized, wherein the neural network to be quantized is a neural network trained based on a target task; The multiple network layers to be quantized are sorted according to the order of their quantization sensitivity. The sorted multiple network layers to be quantized are divided into multiple layer group combinations according to the number of quantization bit widths. Each layer group combination includes multiple network layer groups. According to the quantization sensitivity of the network layer to be quantized in the layer group combination, the quantization bit width is assigned to each network layer group in the multiple layer group combinations to obtain multiple quantization bit width combinations corresponding to the neural network to be quantized, wherein the quantization bit widths corresponding to multiple network layers to be quantized in the network layer group are the same. Based on the inference accuracy of the neural network to be quantized under each of the quantization bit width combinations, the target quantization bit width combination corresponding to the neural network to be quantized is determined from the plurality of quantization bit width combinations.

2. The method for determining the quantization bit width of a neural network according to claim 1, characterized in that, The step of determining the quantization sensitivity of each network layer to be quantized based on the weight matrix of each network layer to be quantized includes: For each network layer to be quantized, the Hessian matrix of the network layer to be quantized is determined, multiple eigenvalues ​​of the Hessian matrix are determined, and the quantization sensitivity corresponding to the network layer to be quantized is determined based on the multiple eigenvalues.

3. The method for determining the quantization bit width of a neural network according to claim 2, characterized in that, Determining the quantization sensitivity of the network layer to be quantized based on multiple feature values ​​includes: The maximum value of the multiple feature values ​​is determined as the quantization sensitivity corresponding to the network layer to be quantized; or... Multiple target values ​​are determined from the plurality of said feature values, and the sum of the plurality of said target values ​​is determined as the quantization sensitivity corresponding to the network layer to be quantized, wherein the plurality of said target values ​​are all greater than the feature values ​​other than the target values ​​among the plurality of said feature values.

4. The method for determining the quantization bit width of a neural network according to claim 1, characterized in that, The step of determining the target quantization bit width combination corresponding to the neural network to be quantized from multiple quantization bit width combinations based on the inference accuracy of the neural network to be quantized under each of the quantization bit width combinations includes: The storage overhead of each of the quantization bit width combinations is determined. Based on the storage overhead of the quantization bit width combination and the inference accuracy of the neural network to be quantized under the quantization bit width combination, a target quantization bit width combination corresponding to the neural network to be quantized is determined from the multiple quantization bit width combinations.

5. The method for determining the quantization bit width of a neural network according to claim 4, characterized in that, The determination of the storage overhead for each of the quantization bit width combinations includes: For each of the quantization bit width combinations, the storage overhead of the quantization bit width combination is determined based on the quantization bit width corresponding to the multiple network layers to be quantized in the quantization bit width combination and the number of weights in the network layers to be quantized.

6. The method for determining the quantization bit width of a neural network according to claim 5, characterized in that, The step of determining the storage overhead of the quantization bit width combination based on the quantization bit width corresponding to the multiple network layers to be quantized in the quantization bit width combination and the number of weights in the network layers to be quantized includes: Determine the quantization bit width corresponding to each of the network layers to be quantized in the quantization bit width combination and the number of weights in the network layer to be quantized; For each of the network layers to be quantized, the product of the quantization bit width and the number of weights corresponding to the network layer to be quantized is determined as the storage overhead of the network layer to be quantized under the combination of quantization bit widths. The sum of the storage overhead of the multiple network layers to be quantized is determined as the storage overhead of the quantization bit width combination.

7. The method for determining the quantization bit width of a neural network according to claim 4, characterized in that, The step of determining the target quantization bit width combination corresponding to the neural network to be quantized from multiple quantization bit width combinations based on the storage overhead of the quantization bit width combination and the inference accuracy of the neural network to be quantized under the quantization bit width combination includes: Based on the storage overhead of the multiple quantization bit width combinations, the quantization bit width combination to be verified is determined from the multiple quantization bit width combinations. Based on the quantization bit width combination to be verified, the multiple network layers to be quantized in the neural network to be quantized are quantized to obtain the neural network to be verified. Based on the neural network to be verified, perform multiple inference operations associated with the target task to obtain multiple network inference results. Determine the inference accuracy of the neural network to be verified based on the multiple network inference results. Determine the target quantization bit width combination corresponding to the neural network to be quantized based on the inference accuracy of the neural network to be verified.

8. The method for determining the quantization bit width of a neural network according to claim 7, characterized in that, The step of determining the quantization bit width combination to be verified among the multiple quantization bit width combinations based on the storage overhead of the multiple quantization bit width combinations includes: The multiple quantization bit width combinations are sorted according to their storage overhead, and the quantization bit width combination to be verified is obtained sequentially from the sorted multiple quantization bit width combinations in ascending order of storage overhead. The step of determining the target quantization bit width combination corresponding to the neural network to be quantized based on the inference accuracy of the neural network to be verified includes: In response to the event that the inference accuracy reaches the expected accuracy threshold, the quantization bit width combination to be verified is determined as the target quantization bit width combination corresponding to the neural network to be quantized.

9. The method for determining the quantization bit width of a neural network according to claim 1, characterized in that, The quantization bit widths of multiple network layer groups are allocated in accordance with preset bit width allocation conditions, wherein the preset bit width allocation conditions include that the quantization bit width allocated to the network layer to be quantized with higher quantization sensitivity is not lower than the quantization bit width allocated to the network layer to be quantized with lower quantization sensitivity.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for determining the quantization bit width of the neural network as described in any one of claims 1-9.

Images