Quantization methods, devices, computer equipment, and storage media for neural networks

Abstract

This disclosure provides a method, apparatus, computer device, and storage medium for quantizing a neural network. The neural network includes multiple blocks, each block including at least one network layer. The method includes: obtaining a target quantization configuration set for a group of blocks in the neural network; filtering a candidate quantization configuration set for another group of blocks in the neural network based on the obtained target quantization configuration set for the group of blocks to obtain a target quantization configuration set for the other group of blocks, such that when the other group of blocks is quantized using the quantization configurations in the target quantization configuration set for the other group of blocks, and the group of blocks is quantized using the quantization configurations in the target quantization configuration set for the group of blocks, the boundary values ​​of the target performance parameters of the neural network do not exceed the preset parameter range corresponding to the target performance parameters; determining the target quantization configuration of the neural network based on the target quantization configurations of each block, and performing quantization processing on the neural network.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for quantizing neural networks. Background Technology

[0002] Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), and other types of neural networks have been widely used in various computer vision fields such as image classification and object detection. Generally speaking, the more layers and parameters a neural network has, the more accurate its output will be, but it will also lead to higher computational complexity and greater memory requirements.

[0003] In some application scenarios, the available resources, such as memory, for neural networks are limited, such as when deploying neural networks on embedded devices. For these scenarios, deploying computationally complex neural networks with high memory requirements is impractical. Therefore, it is necessary to process these neural networks to reduce their memory requirements, enabling their deployment in resource-constrained environments.

[0004] One feasible approach to deploying neural networks in resource-constrained scenarios is to quantize the full-precision parameters of the neural network to a lower precision, thereby reducing memory requirements by using fewer bits to store the parameters. Hardware devices can support various bit-width quantization methods; for example, 8-bit quantization is commonly used in industry to quantize the model parameters of the neural network to an 8-bit representation.

[0005] Because each layer in a neural network has a different sensitivity to quantization—meaning that for the same quantization bit width, different network layers will experience different precision losses after quantization—different network layers can be assigned different quantization bit widths to enable the quantized neural network to output high-precision results even in resource-constrained scenarios. However, since each layer of a neural network has multiple bit widths to choose from, it is often necessary to consider all combinations of each bit width in each layer to select an optimal bit width configuration. For a multi-layer neural network, the number of bit width configuration combinations to consider is large, resulting in relatively low quantization efficiency. Summary of the Invention

[0006] To overcome the problems existing in the related technologies, the present disclosure provides a method, apparatus, computer device and storage medium for quantizing neural networks, so as to solve the defects in the related technologies.

[0007] According to a first aspect of the present disclosure, a quantization method for a neural network is provided. The neural network includes multiple blocks, each block including at least one network layer. The method includes: acquiring a target quantization configuration set for a group of blocks in the neural network, the group of blocks including at least one block; filtering a candidate quantization configuration set for another group of blocks in the neural network based on the acquired target quantization configuration set for the group of blocks to obtain a target quantization configuration set for the other group of blocks, such that when the other group of blocks is quantized using the quantization configurations in the target quantization configuration set for the other group of blocks, and when the group of blocks is quantized using the quantization configurations in the target quantization configuration set for the group of blocks, the boundary values ​​of the target performance parameters of the neural network do not exceed a preset parameter range corresponding to the target performance parameters; determining the target quantization configuration of the neural network based on the target quantization configurations of each block, and performing quantization processing on the neural network.

[0008] Optionally, one set of blocks includes the first j blocks in the neural network, and the other set of blocks includes the first i blocks in the neural network, where i is a positive integer greater than 1 and j is a positive integer less than i.

[0009] Optionally, the step of filtering the candidate quantization configuration set of another block in the neural network based on the acquired target quantization configuration set of one block to obtain the target quantization configuration set of the other block includes: for each quantization configuration in the candidate quantization configuration set of the first i blocks, determining the boundary value of the target performance parameter of the neural network after the first i blocks are quantized by the quantization configuration; if the boundary value of the target performance parameter of the neural network exceeds the preset parameter range corresponding to the target performance parameter, filtering out the quantization configuration from the candidate quantization configuration set of the first i blocks to obtain the target quantization configuration set of the first i blocks.

[0010] Optionally, the method further includes: determining a candidate quantization configuration set for the first i blocks based on the target quantization configuration set for the first j blocks.

[0011] Optionally, determining the candidate quantization configuration set for the first i blocks based on the target quantization configuration set for the first j blocks includes: permuting and combining each quantization configuration in the target quantization configuration set for the first j blocks with each quantization configuration in the initial quantization configuration set for the i-th block of the neural network to obtain the initial quantization configuration set for the first i blocks; for each quantization configuration in the initial quantization configuration set for the first i blocks, determining the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration, wherein the quantization sensitivity parameter corresponding to a network layer is used to characterize the difference between the output result before quantization and the output result after quantization of the network layer; determining the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration; selecting at least one quantization configuration from the initial quantization configuration set for the first i blocks to obtain the candidate quantization configuration set for the first i blocks, wherein the estimated value of the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the estimated value of the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0012] Optionally, determining the estimated value of the quantization sensitivity parameter of the first i blocks under the quantization configuration based on the quantization sensitivity parameter of each network layer in the first i blocks under the quantization configuration includes: summing the quantization sensitivity parameters of each network layer in the first i blocks under the quantization configuration to obtain the estimated value of the quantization sensitivity parameter of the first i blocks under the quantization configuration.

[0013] Optionally, the method further includes: for each quantization configuration in the target quantization configuration set of the first i blocks, determining the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration, wherein the quantization sensitivity parameter corresponding to the first i blocks is used to characterize the difference between the output result before quantization and the output result after quantization of the first i blocks; selecting at least one quantization configuration from the target quantization configuration set of the first i blocks to obtain a new target quantization configuration set of the first i blocks, wherein the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0014] Optionally, determining the quantization sensitivity parameter corresponding to the first i blocks under each quantization configuration in the target quantization configuration set for the first i blocks includes: inputting calibration data into the neural network, obtaining a first output result of the i-th block when the first i blocks are not quantized, and obtaining a second output result of the i-th block when the first i blocks are quantized by the quantization configuration; and determining the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the similarity between the first output result and the second output result.

[0015] Optionally, the step of selecting at least one quantization configuration from the target quantization configuration set of the first i blocks to obtain a new target quantization configuration set for the first i blocks includes: obtaining multiple pre-divided size intervals, where different size intervals correspond to different size ranges after quantization of the neural network, and the size after quantization of the neural network is related to the quantization configuration used by the neural network; for each size interval, determining an interval quantization configuration from the target quantization configuration set of the first i blocks that makes the size of the neural network after quantization of the first i blocks fall within the size interval; selecting at least one target interval quantization configuration from the interval quantization configuration to obtain a new target quantization configuration set for the first i blocks; wherein the quantization sensitivity parameter corresponding to the first i blocks under the target interval quantization configuration is greater than the quantization sensitivity parameter corresponding to the first i blocks under the interval quantization configuration other than the target interval quantization configuration.

[0016] Optionally, the target performance parameter includes at least one of the following: the output accuracy of the quantized neural network; the model size of the quantized neural network; and the memory usage of the quantized neural network.

[0017] According to a second aspect of the present disclosure, a quantization apparatus for a neural network is provided. The neural network includes multiple blocks, each block including at least one network layer. The apparatus includes: an acquisition module, configured to acquire a target quantization configuration set for a group of blocks in the neural network, the group of blocks including at least one block; a filtering module, configured to filter a candidate quantization configuration set for another group of blocks in the neural network based on the acquired target quantization configuration set for the group of blocks, to obtain a target quantization configuration set for the other group of blocks, such that when the other group of blocks is quantized using the quantization configurations in the target quantization configuration set for the other group of blocks, and when the group of blocks is quantized using the quantization configurations in the target quantization configuration set for the group of blocks, the boundary values ​​of the target performance parameters of the neural network do not exceed a preset parameter range corresponding to the target performance parameters; and a quantization module, configured to determine the target quantization configuration of the neural network based on the target quantization configurations of each block, and to perform quantization processing on the neural network.

[0018] Optionally, one set of blocks includes the first j blocks in the neural network, and the other set of blocks includes the first i blocks in the neural network, where i is a positive integer greater than 1 and j is a positive integer less than i.

[0019] Optionally, the filtering module is configured to: for each quantization configuration in the candidate quantization configuration set of the first i blocks, determine the boundary value of the target performance parameter of the neural network after quantization by the quantization configuration of the first i blocks; if the boundary value of the target performance parameter of the neural network exceeds the preset parameter range corresponding to the target performance parameter, filter out the quantization configuration from the candidate quantization configuration set of the first i blocks to obtain the target quantization configuration set of the first i blocks.

[0020] Optionally, the apparatus further includes: a candidate quantization configuration set determination module, used to determine a candidate quantization configuration set for the first i blocks based on the target quantization configuration set for the first j blocks.

[0021] Optionally, the candidate quantization configuration set determination module is configured to: permutate and combine each quantization configuration in the target quantization configuration set of the first j blocks with each quantization configuration in the initial quantization configuration set of the i-th block of the neural network to obtain an initial quantization configuration set for the first i blocks; for each quantization configuration in the initial quantization configuration set of the first i blocks, determine the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration, wherein the quantization sensitivity parameter corresponding to a network layer is used to characterize the difference between the output result before quantization and the output result after quantization of the network layer; based on the quantization sensitivity parameters corresponding to each network layer in the first i blocks under the quantization configuration, determine the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration; and select at least one quantization configuration from the initial quantization configuration set of the first i blocks to obtain a candidate quantization configuration set for the first i blocks, wherein the estimated value of the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the estimated value of the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0022] Optionally, the candidate quantization configuration set determination module is used to: sum the quantization sensitivity parameters corresponding to each network layer in the first i blocks under the quantization configuration to obtain the estimated values ​​of the quantization sensitivity parameters corresponding to the first i blocks under the quantization configuration.

[0023] Optionally, the apparatus further includes: a quantization sensitivity parameter determination module, configured to determine, for each quantization configuration in the target quantization configuration set of the first i blocks, the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration, wherein the quantization sensitivity parameter corresponding to the first i blocks is used to characterize the difference between the output result before quantization and the output result after quantization of the first i blocks; and a reselection module, configured to select at least one quantization configuration from the target quantization configuration set of the first i blocks to obtain a new target quantization configuration set of the first i blocks, wherein the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0024] Optionally, the quantization sensitivity parameter determination module is configured to: input calibration data into the neural network, obtain a first output result of the i-th block when the first i blocks are not quantized, and obtain a second output result of the i-th block when the first i blocks are quantized by the quantization configuration; and determine the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the similarity between the first output result and the second output result.

[0025] Optionally, the reselection module is used to: obtain multiple pre-divided size intervals, where different size intervals correspond to different size ranges after quantization of the neural network, and the size after quantization of the neural network is related to the quantization configuration adopted by the neural network; for each size interval, determine the interval quantization configuration in the target quantization configuration set of the first i blocks that makes the size of the neural network fall within the size interval after quantization of the first i blocks; select at least one target interval quantization configuration from the interval quantization configurations to obtain a new target quantization configuration set for the first i blocks; the quantization sensitivity parameter corresponding to the first i blocks under the target interval quantization configuration is greater than the quantization sensitivity parameter corresponding to the first i blocks under the interval quantization configuration other than the target interval quantization configuration.

[0026] Optionally, the target performance parameter includes at least one of the following: the output accuracy of the quantized neural network; the model size of the quantized neural network; and the memory usage of the quantized neural network.

[0027] According to a third aspect of the present disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a neural network quantization method according to any embodiment of the present disclosure.

[0028] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a method for quantizing a neural network according to any embodiment of the present disclosure.

[0029] When the neural network described in this disclosure performs quantization processing, it uses the target quantization configuration set of one block as prior information to filter the target quantization configuration set of another block. This eliminates the need to select the target quantization configuration of each block from all available quantization configurations, reducing the number of available quantization configurations in the other block, thereby reducing the computational load of subsequent traversals and improving quantization efficiency.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] Figure 1 This is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0032] Figure 2 , Figure 3 , Figure 4 and Figure 5 These are schematic diagrams illustrating the method for determining the target quantization configuration set of the first i blocks in the neural network according to embodiments of this disclosure.

[0033] Figure 6 This is a block diagram of a quantization device for a neural network according to an embodiment of the present disclosure.

[0034] Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0037] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0038] In this disclosure, the quantized neural network comprises multiple blocks, each block including at least one network layer in the neural network. Different blocks may include the same or different numbers of network layers. Since a neural network often contains a large number of layers, searching for the optimal quantization configuration for all layers of the entire neural network requires calculating the quantization sensitivity of all quantization configurations, resulting in a huge computational load, long quantization time, and low efficiency. For example, for a 10-layer neural network, when three quantization bit widths are available, 3... 10 Selecting the optimal quantization configuration from various options involves a significant amount of computation. Therefore, for a neural network to be quantized, the network can be divided into multiple blocks, each containing one or more consecutive network layers. The optimal quantization configuration for each block is then calculated sequentially, ultimately determining the optimal quantization configuration for the entire neural network. In one embodiment, the neural network can be divided into blocks based on its network structure. For example, based on the function of each layer in the network structure, several closely related network layers can be grouped into the same block. In another embodiment, the number of blocks, n, can be preset, with each block containing one or more network layers.

[0039] In one embodiment, one or more intervals can be preset to divide the entire neural network into multiple blocks according to the preset intervals, such that each block contains no more than the number of network layers of the preset interval value. For example, for a neural network with L network layers, the interval can be preset to 'a', dividing the neural network into n blocks, where n is the smallest positive integer not less than L / a, and each block contains at most 'a' network layers. Specifically, if L is divisible by 'a', each block includes 'a' network layers; if L is not divisible by 'a', each block except the last one contains 'a' network layers.

[0040] like Figure 1 As shown, this disclosure provides a method for quantizing a neural network, the neural network comprising multiple blocks, each block comprising at least one network layer in the neural network; the method includes:

[0041] Step S101: Obtain a set of target quantization configurations for a group of blocks in the neural network, wherein the group of blocks includes at least one block.

[0042] Step S102: Based on the acquired target quantization configuration set of a group of blocks, filter the candidate quantization configuration set of another group of blocks in the neural network to obtain the target quantization configuration set of the other group of blocks, such that when the other group of blocks is quantized by the quantization configuration in the target quantization configuration set of the other group of blocks, and when the group of blocks is quantized by the quantization configuration in the target quantization configuration set of the group of blocks, the boundary values ​​of the target performance parameters of the neural network do not exceed the preset parameter range corresponding to the target performance parameters.

[0043] Step S103: Determine the target quantization configuration of the neural network based on the target quantization configuration of each block, and perform quantization processing on the neural network.

[0044] In step S101, the target quantization configuration of the neural network includes the target quantization configuration of each block in the neural network, and the target quantization configuration of a block includes the quantization parameters of each network layer in the block. For example, assuming the neural network includes n blocks, and each block includes a network layers, then for the i-th (1≤i≤n) block, the target quantization configuration of that block can be denoted as: The target quantization configuration of the entire neural network can be denoted as: in, Let represent the quantization parameters of the j-th network layer in the i-th block, where 1 ≤ j ≤ a. Of course, the above is merely an illustrative example; the number of network layers in each block of a neural network may not be equal. Therefore, the target quantization configuration set for a group of blocks can include the target quantization configuration of each network layer in all the blocks included in this group. For example, if a group of blocks includes m blocks, and each block includes a network layers, then this group of blocks includes a total of m × a network layers' worth of target quantization configurations. Here, m is a positive integer less than n.

[0045] In step S102, after obtaining the target quantization configuration set of each group of blocks in the neural network, the candidate quantization configuration set of another group of blocks in the neural network can be filtered based on the obtained target quantization configuration set of the first group of blocks to obtain the target quantization configuration set of the other group of blocks. The other group of blocks also includes at least one block, and the number of blocks included in the first group and the other group of blocks can be the same or different. In some embodiments, the first group of blocks is a subset of the other group of blocks; for example, the first group of blocks includes a first block, and the other group of blocks includes a first block, a second block, and a third block. In other embodiments, the first group of blocks and the other group of blocks partially overlap, that is, the two groups of blocks may include some identical blocks and some different blocks. For example, the first group of blocks may include a second block and a third block, and the other group of blocks may include a third block and a fourth block. In other embodiments, the first group of blocks and the other group of blocks may be completely different. For example, the first group of blocks includes a first block, a second block, and a third block; the other group of blocks includes a fourth block and a fifth block. Here, the i-th block represents the block numbered i. The smaller the block number, the closer the block is to the input of the neural network.

[0046] In this context, each optional quantization configuration in the target quantization configuration set of the other block satisfies the following condition: when the other block is quantized by the optional quantization configuration, and the block is quantized by any quantization configuration in the target quantization configuration set of the block, the boundary values ​​of the target performance parameters of the neural network do not exceed the preset parameter range corresponding to the target performance parameters.

[0047] For example, suppose one set of target quantization configurations includes three quantization configurations: Q1, Q2, and Q3, and another set of target quantization configurations includes two quantization configurations: Q4 and Q5. Then, in any of the following cases, the boundary values ​​of the target performance parameters of the neural network will not exceed the preset parameter range corresponding to the target performance parameters:

[0048] One set of blocks uses quantization configuration Q1, and the other set of blocks uses quantization configuration Q4;

[0049] One set of blocks uses quantization configuration Q2, and the other set of blocks uses quantization configuration Q4;

[0050] One set of blocks uses quantization configuration Q3, and the other set of blocks uses quantization configuration Q4;

[0051] One set of blocks uses quantization configuration Q1, and the other set of blocks uses quantization configuration Q5;

[0052] One set of blocks uses quantization configuration Q2, and the other set of blocks uses quantization configuration Q5;

[0053] One set of blocks uses quantization configuration Q3, and the other set of blocks uses quantization configuration Q5.

[0054] The target quantization configuration set of another block can be obtained by filtering the candidate quantization configuration set of another block based on the above conditions. If a certain quantization configuration in the candidate quantization configuration set of another block does not meet the above conditions, then the quantization configuration is filtered out from the candidate quantization configuration set of another block, and the remaining quantization configurations constitute the target quantization configuration set of another block.

[0055] The target quantization configuration set of a neural network can be determined after iterating through each block of the neural network. In each iteration, a set of blocks can be selected from the neural network, and the performance parameters of the neural network under the quantization configuration of this set of blocks can be used as prior information to determine the target quantization configuration set of another set of blocks in the neural network, until all blocks of the neural network have been traversed.

[0056] For example, the traversal starts from the first block. In the i-th traversal, based on the target quantization configuration set of the first j blocks obtained from the previous traversal and the initial quantization configuration set of the i-th block, the target quantization configuration set of the first i blocks is determined after calculation and filtering. In the first traversal, i.e., when i is 1, the target quantization configuration set of the first j blocks can be considered as an empty set, that is, the target quantization configuration set of the first block is directly calculated and determined based on the initial quantization configuration set of the first block. If the neural network includes n blocks, then after the last traversal, the target quantization configuration set of the first n blocks can be determined, that is, the target quantization configuration set of the neural network. In some embodiments, j equals i-1.

[0057] Of course, the above embodiments are merely illustrative examples. The traversal methods in the embodiments of this disclosure are not limited thereto. For example, the block traversed in the j-th traversal may only include a portion of the block traversed in the i-th traversal. Furthermore, the block traversed in the j-th traversal may be completely different from the block traversed in the i-th traversal.

[0058] In one embodiment, the quantization configuration with the highest quantization sensitivity can be selected as the target quantization configuration of the neural network based on the quantization sensitivity of each quantization configuration in the target quantization configuration set.

[0059] The quantization sensitivity of a neural network represents the difference between the output of the quantized neural network and the output of the unquantized, fully-precision neural network for the same input data; that is, the precision loss of the output of the quantized neural network. Generally, the higher the quantization sensitivity, the smaller the difference between the output of the quantized neural network and the output of the fully-precision neural network, the smaller the precision loss of the quantized neural network's output, and the higher the precision of the quantized neural network's output, which better meets the expected requirements. In one embodiment, the metric for measuring the quantization sensitivity of a neural network can be the cosine similarity between the output of the quantized neural network and the output of the fully-precision neural network. In another embodiment, the metric for measuring the quantization sensitivity of a neural network can also be the KL (Kullback-Leibler) divergence between the output of the quantized neural network and the output of the fully-precision neural network, or other similarity metrics.

[0060] In one embodiment, the quantization configuration that best meets the priority criteria in the target quantization configuration set of the neural network can also be selected as the target quantization configuration of the neural network based on priority criteria. For example, if the priority criterion is the model size of the neural network, the model size of the neural network after quantization by each quantization configuration in the target quantization configuration set of the neural network can be calculated, and the quantization configuration with the smallest model size can be selected as the target quantization configuration.

[0061] In step S103, the neural network can be quantized based on the target quantization configuration. The target quantization configuration may include one or more. In some embodiments, multiple target quantization configurations may be pushed to the user, who can then select one as the final quantization configuration. In other embodiments, one may be selected from multiple target quantization configurations based on other conditions, or randomly selected as the final quantization configuration.

[0062] In this disclosure, the target quantization configuration of the neural network includes the quantization bit width parameter of each network layer in the neural network. Each network layer in the neural network is quantized according to each quantization bit width parameter in the target quantization configuration, so that the quantized neural network can have high accuracy, but does not have the large memory requirements of a full-precision neural network, and can be deployed in application scenarios with limited resources.

[0063] Figure 2 This is a schematic diagram illustrating the method for determining the target quantization configuration set of the first i blocks in a neural network. For example... Figure 2 As shown, the method includes:

[0064] Step S201: Based on the target quantization configuration set of the first j blocks, determine the candidate quantization configuration set of the first i blocks.

[0065] Step S202: For each quantization configuration in the candidate quantization configuration set of the first i blocks, determine the boundary values ​​of the target performance parameters of the neural network after the first i blocks are quantized by the quantization configuration.

[0066] Step S203: When the boundary value of the target performance parameter of the neural network under the quantization configuration exceeds the preset parameter range corresponding to the target performance parameter, the quantization configuration is filtered out from the candidate quantization configuration set of the first i blocks to obtain the target quantization configuration set of the first i blocks.

[0067] Where i and j are both positive integers, and i is greater than j. The following explanation uses the case of j = i-1 as an example to illustrate the scheme of this disclosure. In one embodiment, the candidate quantization configuration set of the first i blocks is equivalent to the initial quantization configuration set of the first i blocks. The initial quantization configuration set of the i-th block is obtained by permuting and combining each quantization bit width parameter of each network layer in the block. The initial quantization configuration set of the first 1 block is equivalent to the initial quantization configuration set of the first 1 block. If i is greater than 1, the initial quantization configuration set of the first i blocks can be obtained by permuting and combining each quantization configuration in the target quantization configuration set of the first i-1 blocks with each quantization configuration in the initial quantization configuration set of the i-th block of the neural network. Assume that each block includes a network layers, and assume that the target quantization configuration set of the first i-1 blocks includes quantization configurations. and quantitative configuration The initial quantization configuration set of the i-th block includes quantization configurations. and quantitative configuration in:

[0068]

[0069]

[0070]

[0071]

[0072] The initial quantization configuration set for the first i blocks can include four quantization configurations, namely:

[0073]

[0074]

[0075]

[0076]

[0077] in, This represents the quantization parameters of the j-th network layer in the i-th block under the k-th quantization configuration. In this disclosure, for the same neural network, the same block, or the same network layer, the initial quantization configuration set represents the set of all initially selectable quantization configurations, containing the largest number of quantization configurations. The candidate quantization configuration set is a subset of the initial quantization configuration set, and the target quantization configuration set is a subset of the candidate quantization configuration set, with the target quantization configurations included in the target quantization configuration set. For example, the target quantization configuration and the target quantization configuration set of a neural network represent one or more superior quantization configurations of the neural network selected after calculation and filtering; the initial quantization configuration set of a neural network represents all permutations and combinations of all selectable quantization configurations of the neural network without calculation and filtering; the candidate quantization configuration set of a neural network represents multiple superior quantization configurations of the neural network without or after partial calculation and filtering. In one embodiment, the candidate quantization configuration set can also be equivalent to the initial quantization configuration set or the target quantization configuration set.

[0078] In this disclosure, when quantizing a neural network, a certain preset parameter range is set for its target performance parameters. This preset parameter range can be interpreted as the minimum requirement of the target performance parameters of the quantized neural network for the application scenario. The target performance parameters of the neural network can be interpreted as performance parameters that meet certain requirements. In one embodiment, the target performance parameters of the neural network may include at least one of the following: the output accuracy of the quantized neural network, the model size of the quantized neural network, and the memory usage of the quantized neural network. For example, if the application scenario requires the output accuracy of the quantized neural network to be at least θ to meet industry standards, then in this example, the target performance parameter is the accuracy of the output result of the quantized neural network, and the preset parameter range is higher than θ. For example, if the application scenario requires the model size of the quantized neural network to not exceed α to enable successful deployment, then in this example, the target performance parameter is the model size of the quantized neural network, and the preset parameter range is less than α. For example, if the application scenario requires the memory usage of the quantized neural network to not exceed β to enable successful deployment, then in this example, the target performance parameter is the memory usage of the quantized neural network, and the preset parameter range is less than β.

[0079] It should be understood that the target performance parameters described in this disclosure include, but are not limited to, the target performance parameters mentioned above. The target performance parameters may also be other required performance parameters, and this disclosure does not impose any limitations on them. In one embodiment, the target performance parameters of the neural network may be a single performance parameter or may include multiple target performance parameters, and this disclosure does not impose any limitations on them.

[0080] In this disclosure, for each quantization configuration in the candidate quantization configuration set of the first i blocks, the boundary value of the target performance parameter of the neural network after quantization by the quantization configuration can be calculated as follows: the first i blocks of the neural network are quantized using the quantization parameters in the quantization configuration, and the other network layers of the neural network are treated as using limit values ​​for quantization. The value of the target performance parameter of the entire neural network after processing is the boundary value of the target performance parameter of the neural network after quantization by the quantization configuration. In one embodiment, if the target performance parameter is the output precision of the neural network after quantization, then the boundary value of the target performance parameter of the neural network after quantization by the quantization configuration can be: the first i blocks of the neural network are quantized using the quantization parameters in the quantization configuration, and the other network layers of the neural network are treated as full-precision network layers, i.e., no quantization is performed. The precision of the output result of the entire neural network after processing is the boundary value of the boundary value of the boundary value of the target performance parameter of the neural network.

[0081] It is worth noting that the boundary value of the target performance parameter after the neural network is quantized by the quantization configuration can be interpreted as either the upper boundary of the target performance parameter after the neural network is quantized by the quantization configuration (i.e., the upper limit of the performance parameter that the neural network can theoretically reach) or the lower boundary of the target performance parameter after the neural network is quantized by the quantization configuration (i.e., the lower limit of the performance parameter that the neural network can theoretically reach), depending on the type of the preset parameter range of the target performance parameter.

[0082] The boundary value of the target performance parameter of the neural network under the quantization configuration exceeds the preset parameter range of the performance parameter corresponding to the target performance parameter. This can be either the upper boundary of the target performance parameter of the neural network under the quantization configuration being less than the preset minimum value of the performance parameter corresponding to the target performance parameter, or the lower boundary of the target performance parameter of the neural network under the quantization configuration being greater than the preset maximum value of the performance parameter corresponding to the target performance parameter.

[0083] For example, the size and / or inference time of a neural network generally have upper boundaries. That is, in practical applications, the size of the neural network is required not to exceed a certain model size threshold, and / or the inference time of the neural network is required not to exceed a preset time. The aforementioned model size threshold and preset time are the upper boundaries. As another example, the output accuracy of a neural network generally has a lower boundary. That is, the output accuracy of the neural network is required not to be lower than a certain accuracy threshold. This accuracy threshold is the lower boundary.

[0084] The purpose of this boundary value is to ensure that if the neural network quantized using this quantization configuration for the first i blocks does not meet the preset target performance parameter requirements, then the quantization configuration of the entire neural network, including this quantization configuration, also does not meet the preset target performance parameter requirements, i.e., it is not suitable as the target quantization configuration. When the target performance parameter is the size of the quantized neural network model, assuming that even if each other network layer uses the minimum quantization bit width for quantization in the first i blocks, the size of the entire neural network will still be greater than the model size threshold, this indicates that the quantization configuration Q is not suitable as the quantization configuration for the first i blocks, and thus the quantization configuration Q can be filtered out from the quantization configurations for the first i blocks.

[0085] For example, if the target performance parameter is the output precision of the quantized neural network, and if the first i blocks are quantized using the quantization configuration Q, even if no other network layer of the neural network is quantized, the precision of the final output result is still lower than the preset precision threshold. This means that the quantization configuration Q is not suitable as the quantization configuration for the first i blocks, and thus the quantization configuration Q can be filtered out from the quantization configuration of the first i blocks.

[0086] In the two embodiments described above, when filtering the quantization configurations of other blocks (e.g., the first i+1 blocks) based on the quantization configurations of the first i blocks, it is no longer necessary to consider the case where the first i blocks use Q as the quantization configuration, thereby reducing the amount of data processing and improving quantization efficiency.

[0087] In one embodiment, the number of target performance parameters of the neural network is two or more. For example, when the number of target performance parameters is two, the accuracy of the output result of the quantized neural network is required to be higher than θ and the model size is required to be smaller than α. When filtering out quantization configurations whose boundary values ​​of target performance parameters exceed the preset parameter range from the candidate quantization configuration set of the first i blocks, two target performance parameters can be considered simultaneously. That is, any quantization configuration whose boundary value of target performance parameter exceeds the corresponding preset parameter range can be filtered out from the candidate quantization configuration set of the first i blocks. In other words, the quantization configurations in the target quantization configuration set must simultaneously satisfy that the boundary values ​​of each target performance parameter do not exceed the corresponding preset parameter range. Taking the case where the number of target performance parameters is two as an example, the quantization configurations in the target quantization configuration set must simultaneously satisfy that the accuracy of the output result of the quantized neural network is higher than θ and the model size is smaller than α, and the target quantization configuration set of the first i blocks is determined based on this condition.

[0088] In one embodiment, the number of target performance parameters of the neural network is 2. When filtering out quantization configurations whose boundary values ​​of the target performance parameters exceed the corresponding preset parameter range from the candidate quantization configuration set of the first i blocks, one of the target performance parameters can be considered first to filter out quantization configurations whose first target performance parameter does not meet the preset threshold corresponding to the first target performance parameter, thereby determining the target quantization configuration set that meets the target performance parameter. Then, for the target quantization configuration set, another target performance parameter is considered to filter out quantization configurations whose second target performance parameter does not meet the preset threshold corresponding to the second target performance parameter, thereby determining the target quantization configuration set that simultaneously meets both target performance parameters. For example, if the output accuracy of the quantized neural network is required to be higher than θ and the model size is smaller than α, we can first calculate the upper limit of the accuracy of the output of each quantization configuration, filter out the quantization configurations whose upper limit of the output accuracy is lower than the accuracy threshold of θ, and obtain the intermediate target quantization configuration set. Then, for each quantization configuration in the intermediate target quantization configuration set, we can calculate the lower limit of the model size of the neural network quantized by that quantization configuration, filter out the quantization configurations whose lower limit of the model size is higher than α, and obtain the quantization configuration that satisfies both target performance parameters at the same time, thereby determining the target quantization configuration set for the first i blocks.

[0089] In one embodiment, the number of target performance parameters of the neural network is greater than two. When filtering out quantization configurations whose upper limit of the target performance parameter is lower than the corresponding performance parameter threshold from the candidate quantization configuration set of the first i blocks, either Scheme 1, which simultaneously filters quantization configurations based on all target performance parameters, or Scheme 2, which sequentially filters quantization configurations based on each target performance parameter, can be used. Furthermore, both methods can be combined, with some quantization configurations filtered based on Scheme 1 and others filtered based on Scheme 2. This disclosure does not impose any limitations on this approach.

[0090] Figure 3 This is a schematic diagram illustrating the method for determining the target quantization configuration set of the first i blocks in a neural network. (And...) Figure 2 The embodiment shown differs in that, after step S203, the quantization configurations in the target quantization configuration set of the first i blocks can be further filtered based on the quantization sensitivity parameter corresponding to each quantization configuration through the following steps:

[0091] Step S301: For each quantization configuration in the target quantization configuration set of the first i blocks, determine the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration.

[0092] Step S302: Select at least one quantization configuration from the target quantization configuration set of the first i blocks to obtain a new target quantization configuration set of the first i blocks. The quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0093] In one embodiment, since the quantization configuration with high output accuracy is often the more expected quantization configuration, when determining the target quantization configuration set for the first i blocks, one or more quantization configurations with the highest quantization sensitivity can be selected from all target quantization configuration sets for the first i blocks, while discarding the remaining quantization configurations with lower quantization sensitivity. This further reduces the number of quantization configurations in the target quantization configuration set for the first i blocks and reduces the number of quantization configurations in the candidate quantization configuration set for the subsequent i+1 blocks, thereby reducing the computational load when traversing the later blocks and achieving more efficient quantization processing.

[0094] In this disclosure, the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration is used to characterize the difference between the output results of the first i blocks before quantization and the output results after quantization. In one embodiment, the quantization sensitivity parameter can be calculated by inputting calibration data into the neural network, obtaining the first output result of the i-th block when the first i blocks are not quantized, and obtaining the second output result of the i-th block when the first i blocks are quantized using the quantization configuration, and determining the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the similarity between the first output result and the second output result. In one embodiment, the similarity between the first output result and the second output result can be represented by their cosine similarity or their KL divergence; this disclosure does not limit this.

[0095] In one embodiment, when selecting at least one quantization configuration with the largest quantization sensitivity parameter from the target quantization configuration set of the first i blocks, the number of selected quantization configurations can be determined based on the requirements or the number of quantization configurations in the target quantization configuration set of the first i blocks during the actual quantization process. This number can be either a positive integer or a percentage, to ensure that the number of quantization configurations in the target quantization configuration set after screening is within a suitable range, so that a certain amount of computation can be reduced, while avoiding the reduction of too many quantization configurations that would prevent the acquisition of the actual optimal quantization configuration.

[0096] Figure 4 This is a schematic diagram illustrating the method for determining the target quantization configuration set of the first i blocks in a neural network. (And...) Figure 2 The embodiment shown differs from the one described above. In the specific implementation of step S201, the quantization configurations in the candidate quantization configuration set of the first i blocks can be filtered based on the estimated value of the quantization sensitivity parameter corresponding to each quantization configuration through the following steps:

[0097] Step S401: Arrange and combine each quantization configuration in the target quantization configuration set of the first i-1 blocks with each quantization configuration in the initial quantization configuration set of the i-th block of the neural network to obtain the initial quantization configuration set of the first i blocks.

[0098] Step S402: For each quantization configuration in the initial quantization configuration set of the first i blocks, determine the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration.

[0099] Step S403: Based on the quantization sensitivity parameters of each network layer in the first i blocks under the quantization configuration, determine the estimated values ​​of the quantization sensitivity parameters of the first i blocks under the quantization configuration.

[0100] Step S404: Select at least one quantization configuration from the initial quantization configuration set of the first i blocks and filter out other quantization configurations to obtain a candidate quantization configuration set for the first i blocks. The estimated value of the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the estimated value of the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0101] In one embodiment, the initial quantization configuration set of the i-th block is obtained by permuting and combining each quantization bit width parameter of each network layer in the i-th block. When i is 1, the target quantization configuration set of the first i-1 blocks can be considered as an empty set, that is, the initial quantization configuration set of the first block is equivalent to the initial quantization configuration set of the first block. If i is greater than 1, the initial quantization configuration set of the first i blocks can be obtained by permuting and combining each quantization configuration in the target quantization configuration set of the first i-1 blocks with each quantization configuration in the initial quantization configuration set of the i-th block of the neural network.

[0102] In this disclosure, the quantization sensitivity parameter corresponding to a network layer is used to characterize the difference between the output result of the network layer before quantization and the output result after quantization. In one embodiment, the quantization sensitivity parameter corresponding to a network layer can be calculated by inputting calibration data into the neural network, obtaining the first output result of the network layer when all network layers are not quantized, and obtaining the second output result of the network layer when only the network layer is quantized by the quantization configuration while other network layers are not quantized. Based on the similarity between the first output result and the second output result, the quantization sensitivity parameter corresponding to the network layer under the quantization configuration is determined. In one embodiment, the similarity between the first output result and the second output result can be represented by their cosine similarity or their KL divergence; this disclosure does not limit this.

[0103] In one embodiment, calculating the quantization sensitivity parameters of the first i blocks requires considering the quantization parameters of each network layer within those i blocks, resulting in high computational complexity. However, calculating the quantization sensitivity parameters of a single network layer only requires considering its quantization parameters, leading to lower computational complexity. Since the first i blocks consist of multiple network layers, it can be assumed that each network layer in the first i blocks is independent of the others, and the quantization parameters of one network layer do not affect the output of the next. Therefore, the estimated value of the quantization sensitivity parameters for the first i blocks under the quantization configuration can be calculated based on the quantization sensitivity parameters of each network layer in the first i blocks under the stated quantization configuration. Since the output of the first i blocks is sequentially processed through each network layer, the precision of the output of each network layer in the first i blocks affects the precision of the output of the first i blocks. The quantization sensitivity parameter of each network layer in the first i blocks also influences the quantization sensitivity parameter of the first i blocks under the given quantization configuration. Therefore, the estimated value of the quantization sensitivity parameter of the first i blocks under the given quantization configuration can be used as a reference when comparing the quantization sensitivity of different quantization configurations. Generally, the higher the estimated value of the quantization sensitivity parameter of the first i blocks under the given quantization configuration, the higher the quantization sensitivity parameter of the first i blocks under the given quantization configuration. Since the estimated values ​​of the quantization sensitivity parameters corresponding to the first i blocks under the quantization configuration have lower computational complexity than the estimated values ​​of the quantization sensitivity parameters corresponding to the first i blocks under the quantization configuration, we can first calculate and compare the estimated values ​​of the quantization sensitivity parameters of each quantization configuration in the initial quantization configuration set of the first i blocks, filter out multiple quantization configurations with higher estimated values ​​of quantization sensitivity parameters, and discard the remaining quantization configurations with lower estimated values ​​of quantization sensitivity parameters. This further reduces the number of quantization configurations in the candidate quantization configuration set of the first i blocks, thereby reducing the computational load of subsequent steps and achieving more efficient quantization processing.

[0104] In one embodiment, since it is assumed that each network layer in the first i blocks is independent of each other, the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration can be obtained by directly summing the quantization sensitivity parameters corresponding to each network layer in the first i blocks under the quantization configuration. Although there is a certain deviation between the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration obtained by direct summation and the actual value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration, the calculation process is simple and has low computational complexity, and can efficiently eliminate some quantization configurations that may have relatively low quantization sensitivity.

[0105] In one embodiment, when selecting the quantization configuration with the largest estimated value of at least one quantization sensitivity parameter from the initial quantization configuration set of the first i blocks, the number of quantization configurations selected can be determined based on the requirements or the number of quantization configurations in the initial quantization configuration set of the first i blocks during the actual quantization process. This number can be either a positive integer or a percentage, to ensure that the number of quantization configurations in the candidate quantization configuration set after screening is within a suitable range, so that a certain amount of computation can be reduced, while avoiding the reduction of too many quantization configurations that would prevent the actual optimal quantization configuration from being obtained.

[0106] In one embodiment, Figure 3 The quantization sensitivity parameter based on quantization configuration shown is used to filter the target quantization configuration set of the first i blocks as the first filter. Figure 4 The filtering of the candidate quantization configuration set for the first i blocks based on the estimated value of the quantization sensitivity parameter is considered as the second filtering. The method for determining the target quantization configuration set for the first i blocks in the neural network can include only the first filtering, only the second filtering, or both; this disclosure does not impose any restrictions on this. The first filtering can be performed before step S202 (i.e., performing the first filtering on the candidate quantization set) or after step S203 (i.e., performing the first filtering on the target quantization set). The second filtering can be performed before step S202 (i.e., performing the second filtering on the candidate quantization set) or after step S203 (i.e., performing the second filtering on the target quantization set). Generally, since the estimated value of the quantization sensitivity parameter has a lower computational cost than the quantization sensitivity parameter itself, if the method for determining the target quantization configuration set for the first i blocks in the neural network includes both the first and second filtering, the second filtering can be performed first, followed by the first filtering. In this way, since some inappropriate quantization configurations have been removed when filtering by the estimated value of the quantization sensitivity parameter, the number of quantization configurations that need to be screened when filtering by the quantization sensitivity parameter can be reduced, thereby reducing the amount of data processing.

[0107] Figure 5 This is a schematic diagram illustrating the method for determining the target quantization configuration set of the first i blocks in a neural network. (And...) Figure 3 Unlike the illustrated embodiment, in the specific implementation of step S302, the following steps can also be used to ensure that the target quantization configuration set of the first i blocks has quantization configurations in different size ranges:

[0108] Step S501: Obtain multiple pre-divided size ranges.

[0109] Step S502: For each size range, determine the range quantization configuration in the target quantization configuration set of the first i blocks that makes the size of the neural network fall into the size range after the first i blocks of the neural network are quantized.

[0110] Step S503: Select at least one target interval quantization configuration from the interval quantization configurations to obtain a new set of target quantization configurations for the first i blocks; the quantization sensitivity parameter of the first i blocks under the target interval quantization configuration is greater than the quantization sensitivity parameter of the first i blocks under the interval quantization configuration other than the target interval quantization configuration.

[0111] In one embodiment, the size of the neural network model before quantization can be calculated and divided into multiple size intervals, with different size intervals corresponding to different size ranges after quantization. The size of the quantized neural network is related to the quantization configuration used by the neural network. Generally, the more low-bit-width quantization parameters used in the quantization configuration, the smaller the size of the quantized neural network; conversely, the more high-bit-width quantization parameters used in the quantization configuration, the larger the size of the quantized neural network. In one embodiment, the size intervals can be divided at equal intervals based on certain numerical values, or they can be divided according to different size requirements; this disclosure does not impose any limitations on this.

[0112] In one embodiment, when determining the target quantization configuration set, for each pre-divided size interval, the interval quantization configuration in the target quantization configuration set that makes the size of the neural network after the first i-th block quantization fall within the size interval can be determined respectively. That is, the quantization configuration set is divided into interval quantization configurations corresponding to different size intervals, and the interval quantization configurations corresponding to different size intervals are filtered and screened based on the quantization sensitivity parameter to ensure that for each size interval, there may be a target interval quantization configuration, so that when there are multiple model size requirements, the optimal target quantization configuration can be directly selected from the corresponding size interval.

[0113] In one embodiment, such as Figure 4In the specific implementation of step S404 in the illustrated embodiment, the scheme of the above embodiment can also be used to ensure that quantization configurations exist in different size intervals in the candidate quantization configuration set of the first i blocks. Specifically, this can be done by obtaining multiple pre-divided size intervals; for each size interval, determining the interval quantization configuration in the initial quantization configuration set of the first i blocks that makes the size of the neural network fall within the size interval after quantization of the first i blocks; then selecting at least one target interval quantization configuration from the interval quantization configurations and filtering out other quantization configurations to obtain the candidate quantization configuration set of the first i blocks; the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the target interval quantization configuration is greater than the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the interval quantization configuration other than the target interval quantization configuration. The explanation and interpretation can be found in the above embodiment, and will not be repeated here.

[0114] In some application scenarios, the neural network is used to process images to be processed. The image to be processed can be input into the neural network, which can use the quantization method described in any of the above embodiments to determine the quantization parameters of each network layer. Based on the determined quantization parameters, the input image, weight parameters, and output image of each network layer are quantized to obtain a quantized input image, quantized weight parameters, and a quantized output image. The quantized output image of each network layer can be used as the input image of the next network layer. The last network layer of the neural network can perform target detection or target recognition on the image to be processed based on the quantized input image of the last network layer. The output images of all network layers except the last network layer can be referred to as feature maps of the image to be processed.

[0115] Corresponding to the aforementioned embodiments of the quantization method for neural networks, this disclosure also provides embodiments of a quantization apparatus for neural networks.

[0116] like Figure 6 As shown, this disclosure provides a quantization device for a neural network, the device comprising:

[0117] The acquisition module 601 is used to acquire a set of target quantization configurations for a group of blocks in the neural network, wherein the group of blocks includes at least one block;

[0118] The filtering module 602 is used to filter the candidate quantization configuration set of another block in the neural network based on the acquired target quantization configuration set of a set of blocks, so as to obtain the target quantization configuration set of the other block, such that when the other block is quantized by the quantization configuration in the target quantization configuration set of the other block, and when the first block is quantized by the quantization configuration in the target quantization configuration set of the first block, the boundary values ​​of the target performance parameters of the neural network do not exceed the preset parameter range corresponding to the target performance parameters.

[0119] The quantization module 603 is used to determine the target quantization configuration of the neural network based on the target quantization configuration of each block, and to perform quantization processing on the neural network.

[0120] Optionally, one set of blocks includes the first j blocks in the neural network, and the other set of blocks includes the first i blocks in the neural network, where i is a positive integer greater than 1 and j is a positive integer less than i.

[0121] Optionally, the filtering module is configured to: for each quantization configuration in the candidate quantization configuration set of the first i blocks, determine the boundary value of the target performance parameter of the neural network after quantization by the quantization configuration of the first i blocks; if the boundary value of the target performance parameter of the neural network exceeds the preset parameter range corresponding to the target performance parameter, filter out the quantization configuration from the candidate quantization configuration set of the first i blocks to obtain the target quantization configuration set of the first i blocks.

[0122] Optionally, the apparatus further includes: a candidate quantization configuration set determination module, used to determine a candidate quantization configuration set for the first i blocks based on the target quantization configuration set for the first j blocks.

[0123] Optionally, the candidate quantization configuration set determination module is configured to: permutate and combine each quantization configuration in the target quantization configuration set of the first j blocks with each quantization configuration in the initial quantization configuration set of the i-th block of the neural network to obtain an initial quantization configuration set for the first i blocks; for each quantization configuration in the initial quantization configuration set of the first i blocks, determine the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration, wherein the quantization sensitivity parameter corresponding to a network layer is used to characterize the difference between the output result before quantization and the output result after quantization of the network layer; based on the quantization sensitivity parameters corresponding to each network layer in the first i blocks under the quantization configuration, determine the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration; and select at least one quantization configuration from the initial quantization configuration set of the first i blocks to obtain a candidate quantization configuration set for the first i blocks, wherein the estimated value of the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the estimated value of the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0124] Optionally, the candidate quantization configuration set determination module is used to: sum the quantization sensitivity parameters corresponding to each network layer in the first i blocks under the quantization configuration to obtain the estimated values ​​of the quantization sensitivity parameters corresponding to the first i blocks under the quantization configuration.

[0125] Optionally, the apparatus further includes: a quantization sensitivity parameter determination module, configured to determine, for each quantization configuration in the target quantization configuration set of the first i blocks, the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration, wherein the quantization sensitivity parameter corresponding to the first i blocks is used to characterize the difference between the output result before quantization and the output result after quantization of the first i blocks; and a reselection module, configured to select at least one quantization configuration from the target quantization configuration set of the first i blocks to obtain a new target quantization configuration set of the first i blocks, wherein the quantization sensitivity parameter corresponding to the selected quantization configuration is greater than the quantization sensitivity parameter corresponding to the unselected quantization configuration.

[0126] Optionally, the quantization sensitivity parameter determination module is configured to: input calibration data into the neural network, obtain a first output result of the i-th block when the first i blocks are not quantized, and obtain a second output result of the i-th block when the first i blocks are quantized by the quantization configuration; and determine the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the similarity between the first output result and the second output result.

[0127] Optionally, the reselection module is used to: obtain multiple pre-divided size intervals, where different size intervals correspond to different size ranges after quantization of the neural network, and the size after quantization of the neural network is related to the quantization configuration adopted by the neural network; for each size interval, determine the interval quantization configuration in the target quantization configuration set of the first i blocks that makes the size of the neural network fall within the size interval after quantization of the first i blocks; select at least one target interval quantization configuration from the interval quantization configurations to obtain a new target quantization configuration set for the first i blocks; the quantization sensitivity parameter corresponding to the first i blocks under the target interval quantization configuration is greater than the quantization sensitivity parameter corresponding to the first i blocks under the interval quantization configuration other than the target interval quantization configuration.

[0128] Optionally, the target performance parameter includes at least one of the following: the output accuracy of the quantized neural network; the model size of the quantized neural network; and the memory usage of the quantized neural network.

[0129] In the above-described apparatus, the functions or modules included in the apparatus provided in this disclosure embodiment can be used to execute the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0130] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0131] This specification also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

[0132] Figure 7 This diagram illustrates a more specific hardware structure of a computing device provided in an embodiment of this specification. The device may include a processor 701, a memory 702, an input / output interface 703, a communication interface 704, and a bus 705. The processor 701, memory 702, input / output interface 703, and communication interface 704 are interconnected internally via the bus 705.

[0133] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor 701 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

[0134] The memory 702 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701.

[0135] The input / output interface 703 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0136] Communication interface 704 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0137] Bus 705 includes a pathway for transmitting information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704).

[0138] It should be noted that although the above-described device only shows the processor 701, memory 702, input / output interface 703, communication interface 704, and bus 705, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0139] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0140] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0141] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this specification, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

[0142] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] The above description is merely a specific implementation of the embodiments of this specification. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this specification, and these improvements and modifications should also be considered within the protection scope of the embodiments of this specification.

Claims

1. A method for quantizing a neural network, the method comprising: The neural network comprises a plurality of blocks, each block comprising at least one network layer in the neural network; the method comprises: obtaining a target quantization configuration set of a group of blocks in the neural network, the group of blocks comprising at least one block; filtering a candidate quantization configuration set of another group of blocks in the neural network based on the obtained target quantization configuration set of the group of blocks, to obtain a target quantization configuration set of the another group of blocks; wherein the target quantization configuration set of the group of blocks and the target quantization configuration set of the another group of blocks satisfy the following condition: in the case that the another group of blocks is quantized by the quantization configurations in the target quantization configuration set of the another group of blocks, and the group of blocks is quantized by the quantization configurations in the target quantization configuration set of the group of blocks, the boundary values of the target performance parameters of the neural network do not exceed the preset parameter range corresponding to the target performance parameters; determining a target quantization configuration of the neural network based on the target quantization configuration of each block, and performing quantization processing on the neural network; The neural network is used for processing a to-be-processed image, and the neural network determines quantization parameters of each network layer by using the quantization method, and performs quantization processing on input images, weight parameters and output images of each network layer based on the determined quantization parameters, to obtain quantized input images, quantized weight parameters and quantized output images.

2. The method of claim 1, wherein, The group of blocks comprises the first j blocks in the neural network, and the another group of blocks comprises the first i blocks in the neural network, i is a positive integer greater than 1, and j is a positive integer less than i.

3. The method of claim 2, wherein, The filtering of the candidate quantization configuration set of the another group of blocks in the neural network based on the target quantization configuration set of the obtained group of blocks comprises: For each quantization configuration in the candidate quantization configuration set of the first i blocks, determining the boundary value of the target performance parameter of the neural network after the first i blocks are quantized by the quantization configuration; In the case that the boundary value of the target performance parameter of the neural network exceeds the preset parameter range corresponding to the target performance parameter, filtering out the quantization configuration from the candidate quantization configuration set of the first i blocks to obtain the target quantization configuration set of the first i blocks.

4. The method of claim 3, wherein, The method further comprises: determining the candidate quantization configuration set of the first i blocks based on the target quantization configuration set of the first j blocks.

5. The method of claim 4, wherein, The determination of the candidate quantization configuration set of the first i blocks based on the target quantization configuration set of the first j blocks comprises: performing permutation and combination on each quantization configuration in the target quantization configuration set of the first j blocks and each quantization configuration in an initial quantization configuration set of the i-th block of the neural network, to obtain an initial quantization configuration set of the first i blocks; For each quantization configuration in the initial quantization configuration set of the first i blocks, determining a quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration, the quantization sensitivity parameter corresponding to a network layer being used to represent the difference between the output result before quantization and the output result after quantization of the network layer. determining an estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration; selecting at least one quantization configuration from the initial set of quantization configurations of the first i blocks to obtain a candidate set of quantization configurations of the first i blocks, the estimated value of the quantization sensitivity parameter corresponding to the selected quantization configuration being greater than the estimated value of the quantization sensitivity parameter corresponding to the unselected quantization configuration.

6. The method of claim 5, wherein, The method further comprises: determining the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration; 7. The method of claim 3, wherein, The method further comprises: determining the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration; The method further comprises:

8. The method of claim 7, wherein, determining the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration; The method further comprises: determining the estimated value of the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the quantization sensitivity parameter corresponding to each network layer in the first i blocks under the quantization configuration; 9. The method of claim 7, wherein, The method further comprises: inputting the calibration data into the neural network to obtain a first output result of the i-th block in the case that the first i blocks are not quantized, and to obtain a second output result of the i-th block in the case that the first i blocks are quantized under the quantization configuration; determining the quantization sensitivity parameter corresponding to the first i blocks under the quantization configuration based on the similarity between the first output result and the second output result. The method further comprises:

10. The method according to any one of claims 1 to 9, characterized in that, obtaining a plurality of size intervals divided in advance, different size intervals corresponding to different size ranges of the neural network after quantization, the size of the neural network after quantization being related to the quantization configuration adopted by the neural network; for each size interval, determining an interval quantization configuration in the target set of quantization configurations of the first i blocks that causes the size of the neural network after quantization of the first i blocks to fall into the size interval; selecting at least one target interval quantization configuration from the interval quantization configurations to obtain the target set of quantization configurations of the first i blocks again; the quantization sensitivity parameter corresponding to the first i blocks under the target interval quantization configuration being greater than the quantization sensitivity parameter corresponding to the first i blocks under the interval quantization configurations other than the target interval quantization configuration. The target performance parameter comprises at least one of: Output precision of the neural network after quantization; Model size of the neural network after quantization; Memory occupation of the neural network after quantization.

11. A quantization apparatus of a neural network, characterized by, The neural network comprises a plurality of blocks, each block comprising at least one network layer in the neural network; and the apparatus comprises: an acquisition module configured to acquire a target quantization configuration set of a group of blocks in the neural network, the group of blocks comprising at least one block; a filtering module configured to filter a candidate quantization configuration set of another group of blocks in the neural network based on the target quantization configuration set of the group of blocks, to obtain a target quantization configuration set of the another group of blocks; wherein the target quantization configuration set of the group of blocks and the target quantization configuration set of the another group of blocks satisfy the following condition: in a case that the another group of blocks is quantized by the quantization configurations in the target quantization configuration set of the another group of blocks, and the group of blocks is quantized by the quantization configurations in the target quantization configuration set of the group of blocks, boundary values of a target performance parameter of the neural network do not exceed a preset parameter range corresponding to the target performance parameter; a quantization module configured to determine a target quantization configuration of the neural network based on the target quantization configuration of each block, and to perform quantization processing on the neural network; The neural network is used to process a to-be-processed image, and the neural network determines quantization parameters of each network layer by using the quantization apparatus, and performs quantization processing on input images, weight parameters and output images of each network layer based on the determined quantization parameters, to correspondingly obtain quantized input images, quantized weight parameters and quantized output images.

12. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 10.

13. A computer readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps of the method of any one of claims 1 to 10.

Images