Neural network-based weight quantization method and apparatus, and electronic device

Abstract

The present disclosure provides a neural network-based weight quantization method, device and electronic equipment. The method comprises: obtaining calibration data corresponding to a calibration data set of the neural network; obtaining first quantized weights corresponding to a preset first truncation ratio of original weights of the neural network, determining a first quantization error of the first quantized weights relative to the original weights according to the calibration data; in the case where the first quantization error is determined to satisfy a preset convergence condition according to the calibration data, performing truncation processing and quantization processing on the original weights of the neural network according to the first truncation ratio to obtain network weights of the neural network. This method can automatically determine an adjusted first truncation ratio, so that the error value of the network weights of the neural network is smaller.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a weight quantization method, apparatus and electronic device based on neural networks. Background Technology

[0002] A neural network is a computational model consisting of a large number of interconnected nodes (neurons) that can simulate the neural network of the human brain for information processing.

[0003] A neural network typically consists of key components such as neurons, layers, weights and biases, and activation functions. Neurons are the basic units, and layers are composed of multiple neurons, usually including an input layer, hidden layers, and an output layer. Weights and biases are important parameters for adjusting the network's behavior.

[0004] Since the weight data in neural networks is usually large, in order to improve the operating efficiency of neural networks, it is necessary to process the weight data to reduce its volume and improve the processing efficiency of the network model. Summary of the Invention

[0005] This disclosure provides a weight quantization method, apparatus, and electronic device based on neural networks.

[0006] In a first aspect, this disclosure provides a weight quantization method based on a neural network, the method comprising:

[0007] Obtain the calibration data corresponding to the calibration dataset for the neural network;

[0008] Obtain the original weights of the neural network corresponding to a first quantized weight with a preset first truncation ratio, and determine the first quantization error of the first quantized weight relative to the original weight based on the calibration data;

[0009] If the calibration data determines that the first quantization error meets the preset convergence condition, the original weights of the neural network are truncated and quantized according to the first truncation ratio to obtain the network weights of the neural network.

[0010] Secondly, this disclosure provides a weight quantization device based on a neural network, the device comprising:

[0011] The acquisition module is adapted to acquire calibration data of the neural network corresponding to the calibration dataset;

[0012] The determination module obtains the first quantized weight corresponding to the preset first truncation ratio of the original weights of the neural network, and determines the first quantization error of the first quantized weight relative to the original weights based on the calibration data;

[0013] The processing module is adapted to, when it is determined from the calibration data that the first quantization error meets a preset convergence condition, truncate and quantize the original weights of the neural network according to the first truncation ratio to obtain the network weights of the neural network.

[0014] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0015] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described method when executed by a processor / processor core.

[0016] Fifthly, this disclosure also provides a many-core chip for running a neural network comprising multiple network segments, wherein the neural network comprises multiple network segments, and the many-core chip comprises multiple processing cores corresponding to the multiple network segments;

[0017] In this process, any one of the processing kernels is used to run the corresponding network segment in the neural network; and, the processing kernel performs weight quantization processing on the weights of multiple operator weights corresponding to multiple target linear operators in the corresponding network segment based on the above method.

[0018] In the embodiments provided in this disclosure, firstly, a first quantization error is determined relative to the original weights based on the first quantization weights corresponding to a preset first truncation ratio of the original weights of the neural network. Correspondingly, if the first quantization error satisfies a preset convergence condition based on calibration data, the network weights of the neural network are obtained according to the first truncation ratio. Therefore, this method can automatically determine whether the first truncation ratio is appropriate based on the preset convergence condition, so that the error value of the network weights of the neural network quantized according to the first truncation ratio is smaller, thereby helping to improve the accuracy of the quantization results and improve the processing efficiency of the model.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 A flowchart illustrating a weight quantization method based on a neural network provided in an embodiment of this disclosure is shown.

[0022] Figure 2 This diagram illustrates outlier values ​​in the weights.

[0023] Figure 3 A flowchart illustrating a neural network-based weight quantization method provided in this disclosure is shown as an example.

[0024] Figure 4 A block diagram of a weight quantization device based on a neural network provided in an embodiment of this disclosure;

[0025] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0028] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0030] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0031] Figure 1 A flowchart illustrating a weight quantization method based on a neural network provided in an embodiment of this disclosure is shown. Figure 1 As shown, the quantification method includes:

[0032] Step S110: Obtain calibration data for the neural network corresponding to the calibration dataset.

[0033] In this context, a calibration dataset can be used to adjust and optimize model parameters during the compression process to reduce performance loss. Calibration datasets are typically not used for training but rather to capture the distribution and features of the input data, enabling the compressed model to better approximate the behavior of the uncompressed model. Calibration datasets can adjust for errors that may be introduced during compression, ensuring that the model maintains high accuracy and performance after compression.

[0034] Based on the calibration dataset, the inference process of the neural network model is executed to obtain calibration data corresponding to the calibration dataset. The calibration data refers to the data results obtained during the operation of the neural network model based on the calibration dataset, specifically including intermediate calculation results, input results, and output results. The calibration data can be obtained through hook functions, callback functions, etc., and is used to calculate quantization error in subsequent steps.

[0035] Step S120: Obtain the first quantization weight corresponding to the preset first truncation ratio of the original weight of the neural network, and determine the first quantization error of the first quantization weight relative to the original weight based on the calibration data.

[0036] The original weights of the neural network can be the operator weights of one or more operators in the neural network, or the overall weights of any network layer in the neural network. This application does not limit the specific form of the original weights.

[0037] The original weights corresponding to the preset first truncation ratio can be obtained in the following way: First, the original weights are truncated according to the preset first truncation ratio, for example, outliers in the original weights can be truncated; then, the truncated original weights are quantized to obtain the first quantized weights.

[0038] In addition, the first quantization error can be used to measure the degree of deviation of the first quantization weight from the original weight. If the first quantization error is larger, it means that the first quantization weight deviates from the original weight, that is, the quantization loss is greater; if the first quantization error is smaller, it means that the first quantization weight deviates from the original weight, that is, the quantization loss is smaller.

[0039] The calibration data can be used as a parameter to calculate the quantization error. Specifically, the original output data can be obtained based on the first operation result (such as multiplication) between the calibration data and the original weights; the quantized output data can be obtained based on the preset operation result between the calibration data and the first quantization weights; and the first quantization error can be obtained based on the second operation result (such as difference operation, norm operation, etc.) between the quantized output data and the original output data.

[0040] Step S130: If the first quantization error is determined to meet the preset convergence condition based on the calibration data, the original weights of the neural network are truncated and quantized according to the first truncation ratio to obtain the network weights of the neural network.

[0041] The preset convergence condition can be used to determine whether the first quantization error meets the business requirements. In specific implementation, if the first quantization error obtained initially does not meet the preset convergence condition, the first truncation ratio can be adjusted to obtain the second quantization error (i.e. the adjusted first quantization error) based on the adjusted first truncation ratio, and then determine whether the second quantization error meets the preset convergence condition.

[0042] In practice, the preset convergence conditions can be set flexibly. For example, it can be set to judge based on the number of times the first truncation ratio is adjusted. If the number of times the first truncation ratio is adjusted reaches a preset threshold, it is determined that the quantization error meets the preset convergence conditions. Alternatively, it can be judged based on the difference between the quantization error corresponding to the first truncation ratio after each adjustment and the quantization error corresponding to the first truncation ratio before adjustment. If the difference is less than a preset value, it is determined that the quantization error meets the preset conditions.

[0043] Furthermore, the preset convergence condition can take other forms. For example, the quantization error can be determined to meet the preset convergence condition if the first quantization error is less than a preset threshold. Specifically, a preset threshold can be set in advance, and the quantization error can be compared with the preset threshold. If the quantization error is greater than or equal to the preset threshold, the preset convergence condition is not met; if the quantization error is less than the preset threshold, the quantization error is determined to meet the preset convergence condition. In summary, this application does not limit the specific form of the preset convergence condition.

[0044] Therefore, this method can automatically determine whether the first truncation ratio is appropriate based on the preset convergence condition, so that the error value of the network weights of the neural network obtained by quantization based on the first truncation ratio is smaller, thereby helping to improve the accuracy of the quantization results and improve the processing efficiency of the model.

[0045] Furthermore, those skilled in the art can make various modifications and variations to the embodiments of this disclosure: In one optional implementation, the original weights of the neural network may include at least one of the following: a weight matrix corresponding to a preset operator in the neural network, at least one matrix unit in the weight matrix corresponding to the preset operator in the neural network, and at least one matrix subunit in the at least one matrix unit. The preset operator in the neural network can be a single operator (also called an independent operator or a non-joint operator), or multiple operators that are related to each other (also called a joint operator). Therefore, this application can perform separate quantization processing on a single weight corresponding to a single operator, or perform collaborative quantization processing on a set of weights corresponding to multiple operators. In addition, the quantization granularity during quantization processing can also be flexibly set: For example, when the original weights of the neural network are the weight matrix corresponding to the preset operator in the neural network, quantization processing is performed at the weight matrix as the granularity. Correspondingly, the preset first truncation ratio can be the first matrix truncation ratio corresponding to the weight matrix corresponding to the network operator in the neural network, to achieve overall quantization processing of the weight matrix. For example, when the original weights of the neural network are at least one matrix unit in the weight matrix corresponding to a preset operator in the neural network, quantization is performed at the matrix unit level. Correspondingly, the preset first truncation ratio can be the first unit truncation ratio corresponding to at least one matrix unit in the weight matrix, to achieve quantization processing for a specific row or column of the weight matrix. Similarly, when the original weights of the neural network are at least one sub-matrix of the aforementioned matrix units, quantization is performed at the granularity of at least one sub-matrix of the matrix units. Correspondingly, the preset first truncation ratio can be the first sub-unit truncation ratio corresponding to at least one sub-matrix of the matrix units, to achieve quantization processing for a portion of elements in a specific row or column of the weight matrix. For example, when the matrix row includes 90 row elements, every 30 row elements can be divided into a matrix sub-unit, thus performing quantization processing at a granularity of every 30 row elements.

[0046] Therefore, the quantization granularity of the weight quantization method in this application can be flexibly selected according to the actual scenario. For example, the quantization granularity can be set to a smaller size (such as quantization in the unit of matrix sub-units), which facilitates quantization processing based on the common characteristics between the same matrix sub-units and improves quantization accuracy. Alternatively, the quantization granularity can be set to a larger size (such as quantization in the unit of weight matrix), which improves quantization speed. The specific configuration can be flexibly configured according to actual business needs.

[0047] In one optional implementation, to accurately determine the truncation ratio, after determining the first quantization error of the first quantization weight relative to the original weight based on calibration data, the following operations can be further performed: if the first quantization error does not meet a preset convergence condition based on calibration data, the first truncation ratio is adjusted according to the first quantization error to obtain an adjusted first truncation ratio; a second quantization weight (also called the adjusted first quantization weight) corresponding to the original weight is obtained, and a second quantization error of the second quantization weight relative to the original weight is determined based on calibration data. Correspondingly, if the second quantization error meets the preset convergence condition based on calibration data, the original weights of the neural network are truncated and quantized according to the adjusted first truncation ratio. This method improves the rationality of the truncation ratio by adjusting the first truncation ratio and making judgments based on the quantization error obtained after the adjustment. Since the first quantization error is used to measure the degree of deviation of the first quantization weight relative to the original weight, an adjustment method can be determined based on the first quantization error, and the first truncation ratio can be adjusted based on the determined adjustment method to obtain the adjusted first truncation ratio. Correspondingly, the second quantization weight corresponding to the adjusted first cutoff ratio can be obtained as follows: The original weight is truncated according to the adjusted first cutoff ratio, and then quantized to obtain the second quantization weight. Thus, the second quantization error can be used to measure the degree of deviation of the second quantization weight from the original weight. The determination method of the second quantization error relative to the original weight is similar to that of the first quantization error. For example, quantization output data can be obtained based on the preset calculation result between calibration data and the second quantization weight; the second quantization error can be obtained based on the second calculation result between the quantization output data and the original output data mentioned above. Correspondingly, the aforementioned preset convergence condition can be used to determine whether the second quantization error meets business requirements. In specific implementation, multiple second cutoff ratios and multiple second quantization weights can be generated, resulting in multiple second quantization errors. Based on the calculation results (such as average, minimum, etc.) between multiple second quantization errors, it can be determined which second quantization error meets the preset convergence condition. Alternatively, it can also be determined whether the second quantization error obtained this time meets the preset convergence condition based on the calculation result between the second quantization error obtained this time and the second quantization error obtained last time. In summary, this application does not limit the specific meaning and judgment method of the preset convergence condition, as long as it can be used to screen the second quantification error with high business value.

[0048] In one optional implementation, when adjusting the first truncation ratio based on the first quantization error to obtain an adjusted first truncation ratio, and then obtaining the second quantization weight corresponding to the adjusted first truncation ratio for the original weights, this can be achieved as follows: The first truncation ratio is adjusted multiple times based on the first quantization error to obtain multiple adjusted first truncation ratios; the second quantization weights corresponding to the original weights for each of the multiple adjusted first truncation ratios are obtained, resulting in multiple second quantization weights corresponding to the multiple adjusted first truncation ratios. Correspondingly, it can be determined whether the second quantization error of the second quantization weight relative to the original weights satisfies a preset convergence condition as follows: For any one of the multiple second quantization weights, the second quantization error of the second quantization weight relative to the original weights is determined based on calibration data, resulting in multiple second quantization errors corresponding to the multiple second quantization weights; based on a preset calculation result between the multiple second quantization errors, the second quantization error satisfying the preset convergence condition is determined from the multiple second quantization errors. The adjusted first truncation ratio corresponding to the second quantization error satisfying the preset convergence condition is used for truncation and quantization processing of the original weights. The preset calculation result between multiple second quantization errors can be the result of the minimum value calculation, the result calculated according to the preset loss function, etc., and this application does not limit it.

[0049] In one optional implementation, the first truncation ratio is adjusted according to a preset ratio range. Accordingly, the preset first truncation ratio can be any truncation ratio within the preset ratio range. Multiple adjusted first truncation ratios can be obtained by: adjusting the first truncation ratio multiple times based on multiple truncation ratios different from the first truncation ratio contained within the preset ratio range, resulting in multiple adjusted first truncation ratios. Optionally, the preset ratio range can include: a first interval endpoint (e.g., the endpoint corresponding to the minimum value in the interval) and a second interval endpoint (e.g., the endpoint corresponding to the maximum value in the interval), and the preset first truncation ratio can be a truncation ratio located at the first interval endpoint within the preset ratio range. Accordingly, multiple adjusted first truncation ratios can be obtained by: traversing multiple truncation ratios within the preset ratio range along the direction from the first interval endpoint to the second interval endpoint, and sequentially determining each traversed truncation ratio as an adjusted first truncation ratio, resulting in multiple adjusted first truncation ratios. Furthermore, the preset calculation result includes: the calculation result of taking the minimum value; then the second quantization error satisfying the preset convergence condition is the second quantization error with the smallest error value among multiple second quantization errors. Therefore, in this method, the second quantization weights can be determined as various ratio intervals within a preset ratio range. By iterating and performing calculations sequentially, a better truncation ratio is selected from the preset ratio intervals to truncate the original weights of the neural network. This method can select the second quantization error with the smallest error value among multiple second quantization errors as the second quantization error that satisfies the preset convergence condition.

[0050] In one optional implementation, multiple adjusted first cutoff ratios can be obtained by: generating an i-th random perturbation value, adjusting the first cutoff ratio i-th time based on the i-th random perturbation value, and obtaining an i-th set of adjusted first cutoff ratios corresponding to the i-th random perturbation value. Since the value of i can be any natural number, multiple sets of adjusted first cutoff ratios can be obtained through multiple adjustments. Each set of adjusted first cutoff ratios can include one or more ratio values. Correspondingly, multiple second quantization errors can include: an i-th set of second quantization errors determined based on calibration data relative to the original weights; wherein, the i-th set of second quantization weights can be the quantization weights of the original weights corresponding to the i-th set of adjusted first cutoff ratios. When determining the second quantization error that satisfies the preset convergence condition from multiple second quantization errors based on the preset calculation results between multiple second quantization errors, this can be achieved by: calculating the error change of the i-th set of second quantization errors relative to the (i-1)-th set of second quantization errors, and judging whether the i-th set of second quantization errors satisfies the preset convergence condition based on the error change; wherein, i is a natural number. For example, a convergence function (such as a loss function) can be set to calculate whether the change in error of the second quantization error of the i-th group relative to the second quantization error of the (i-1)-th group has converged. If the change in error of the second quantization error of the i-th group relative to the second quantization error of the (i-1)-th group is less than a preset value, then the second quantization error of the i-th group is considered to meet the business requirements.

[0051] Optionally, the first truncation ratio adjusted in the i-th group may include: a first sub-ratio corresponding to the first operation (e.g., addition) and a second sub-ratio corresponding to the second operation (e.g., subtraction). Accordingly, when adjusting the first truncation ratio for the i-th time based on the i-th random perturbation value to obtain the first truncation ratio adjusted in the i-th group corresponding to the i-th random perturbation value, this can be achieved as follows: performing a first operation on the first truncation ratio based on the i-th random perturbation value to obtain the first sub-ratio corresponding to the first operation; and performing a second operation on the first truncation ratio based on the i-th random perturbation value to obtain the second sub-ratio corresponding to the second operation. Furthermore, the second quantization error in the i-th group can be set as the minimum value between the first sub-error corresponding to the first sub-ratio and the second sub-error corresponding to the second sub-ratio. The calculation methods for the first sub-error corresponding to the first sub-ratio and the second sub-error corresponding to the second sub-ratio are similar to those mentioned above for calculating the first quantization error or the second quantization error, and will not be repeated here. Furthermore, if the second quantization error of the i-th group does not meet the preset convergence condition, a further (i+1)-th random perturbation value is generated. Based on this (i+1)-th random perturbation value, the first truncation ratio is adjusted (i+1)-th time to obtain the (i+1)-th adjusted first truncation ratio corresponding to the (i+1)-th random perturbation value. This method, by setting random perturbation values, can quickly determine the convergence direction, thus avoiding the problem of numerous attempts in the traversal method. It can determine a better solution after fewer adjustments, improving solution efficiency.

[0052] Specifically, when the original weights of the neural network are the weight matrices corresponding to preset operators in the neural network, the preset operators in the neural network can be linear operators, and the calibration data of the neural network corresponding to the calibration dataset can be the calibration input data and / or calibration output data corresponding to the linear operators. Alternatively, when the preset operators in the neural network include multiple target linear operators of the neural network, the calibration data of the neural network corresponding to the calibration dataset can be the calibration input data corresponding to the target linear operator used as the network input end, and / or the calibration output data corresponding to the target linear operator used as the network output end.

[0053] In one alternative implementation, Figure 1 The weight quantization method shown can be applied individually to any linear operator in a neural network. Correspondingly, the calibration data mentioned above can be the calibration input data and / or calibration output data of any of the multiple linear operators. Furthermore, the preset first truncation ratio can be used to truncate the original weights of any operator, and the network weights of the neural network are the operator weights of any operator. Therefore, it can be seen that in this method, each linear operator in the neural network can be based on... Figure 1The weight quantization method shown obtains the quantized operator weights of the linear operator. This method determines the operator weights of multiple linear operators independently and in parallel, which reduces the computational load of determining individual operator weights and improves the efficiency of determining individual operator weights.

[0054] In one optional implementation, the preset operators are multiple target linear operators in the neural network, and the calibration data are the calibration input data and / or calibration output data of the multiple target linear operators used as network input and output terminals. The network input and output terminals can include: data input terminals and / or data output terminals in the neural network; specifically, they can be the data input or data output terminals of the entire neural network, or the data input or data output terminals of a specific module in a certain layer of the network.

[0055] For example, calibration data refers to the calibration input data corresponding to the target linear operators used as network inputs and / or the calibration output data corresponding to the target linear operators used as network outputs. Specifically, multiple target linear operators collectively constitute an entire neural network, or a specified network layer or segment within the neural network. Correspondingly, the network input or output can refer to the entire neural network or a specific network layer or segment. This method aims to treat multiple target linear operators in the neural network as a whole and jointly determine the weight quantization parameters. The preset first truncation ratio can be represented by a first truncation ratio vector, which may include multiple first vector elements corresponding to the multiple target linear operators. Each first vector element can be used to truncate the original weights of the corresponding target linear operators in the neural network. Furthermore, the original weights of the neural network include multiple original weights corresponding to the multiple target linear operators in the neural network; the network weights of the neural network may include multiple operator weights corresponding to the multiple target linear operators in the neural network. Accordingly, the first quantization weight includes: multiple first quantization weights corresponding to multiple original weights, wherein any first quantization weight can be determined in the following way: for any original weight corresponding to any linear operator in the neural network, select the first vector element corresponding to any linear operator from multiple first vector elements of the first truncation ratio vector, and obtain the first quantization weight corresponding to any linear operator based on the selected first vector element.

[0056] It should be noted that, for simplicity, the above description only assumes the original weights of the neural network are the weight matrix corresponding to the preset operators in the neural network. In reality, when the preset operators are multiple target linear operators in the neural network, the original weights of the neural network can also be at least one matrix unit or at least one sub-unit of the weight matrix corresponding to the preset operators. Correspondingly, each first vector element in the first truncation ratio vector can be used to truncate at least one matrix unit or at least one sub-unit of the weight matrix of the corresponding target linear operator in the neural network. Optionally, the first quantization error includes multiple first quantization errors corresponding to the multiple original weights; the adjusted first truncation ratio is represented by the adjusted first truncation ratio vector, which includes multiple second vector elements corresponding to the multiple linear operators. Furthermore, when adjusting the first truncation ratio based on the first quantization error to obtain the adjusted first truncation ratio, this can be achieved in the following way: performing a preset error calculation (such as averaging or finding the maximum or minimum value) on multiple first quantization errors to obtain the processed first quantization error; and coordinating the adjustment of multiple first vector elements of the first truncation ratio vector based on the processed first quantization error to obtain multiple second vector elements of the adjusted first truncation ratio vector.

[0057] For example, assuming the original weights of a neural network are N weight matrices corresponding to N objective linear operators in the neural network, the first truncation ratio vector can be generated as follows: Obtain the ratio information of N first truncation ratios corresponding one-to-one with the N weight matrices, and concatenate the ratio information of the N first truncation ratios to obtain the first truncation ratio vector. Thus, multiple vector element values ​​in the first truncation ratio vector correspond to the ratio information of the N first truncation ratios. To obtain the initial ratio of all weights or weight rows of a model (using the same method), in the above method, multiple ratios corresponding to each weight matrix or each row of the weight matrix can be concatenated into a vector to obtain the variable to be solved (i.e., the first truncation ratio vector). Correspondingly, the coordinated adjustment of the N first truncation ratios can be achieved. Where each weight matrix corresponds to the same truncation ratio, the ratio information of each first truncation ratio can be a ratio value; correspondingly, the ratio values ​​of multiple first truncation ratios are concatenated to form the first truncation ratio vector. When different rows or columns in each weight matrix correspond to different cutoff ratios, the ratio information of each first cutoff ratio can be a small vector containing multiple ratio values, and each ratio value corresponds to a specified row or column in the weight matrix. Accordingly, the vectors (smaller vectors) corresponding to multiple first cutoff ratios are concatenated to form the first cutoff ratio vector (larger vector) mentioned above.

[0058] In the coordinated adjustment process, multiple first vector elements of the first truncation ratio vector can be treated as a whole and adjusted in a coordinated manner. This avoids the error accumulation problem caused by adjusting the weight matrix corresponding to a single operator individually, thus improving the quantization accuracy after adjustment. The inventors discovered that when adjusting the weight matrix corresponding to each operator individually, since the weight matrices of each operator are adjusted independently, if one weight matrix is ​​adjusted inaccurately, resulting in an error, the weight matrices of other operators will not be aware of this error during the adjustment process, potentially leading to error accumulation and an inaccurate final weight matrix. In the coordinated adjustment process, since the multiple weight matrices corresponding to multiple target linear operators are adjusted together, even if an error occurs, the error will be evenly distributed across the various weight matrices, achieving coordinated adjustment among multiple weight matrices and thus avoiding large error accumulation problems.

[0059] In one optional implementation, the neural network may include multiple network segments, each network segment including multiple linear operators, and the calibration data is the calibration input data and / or calibration output data of the operators used as segment input and / or output terminals in any network segment. The segment input and output terminals can be the data input terminals and / or data output terminals of the corresponding network segments. The network segments can be divided according to network layers or network functional modules, specifically a network layer or a network functional module in the neural network. Furthermore, the first truncation ratio vector includes multiple vector elements corresponding to the multiple linear operators in any network segment, each vector element being used to truncate the original weights of the corresponding linear operators in any network segment; thus, the network weights of the neural network include multiple operator weights corresponding to the multiple linear operators in any network segment. Therefore, there is a mapping relationship between the multiple vector elements in the first truncation ratio vector and the multiple linear operators in the network segments, enabling the truncation of the weight data corresponding to each linear operator based on this mapping relationship. In this approach, by dividing the neural network into multiple network segments, weights can be adjusted for each segment separately. Furthermore, the weight adjustment process for multiple network segments can be implemented in parallel. This approach avoids error accumulation within a single network segment while improving the efficiency of weight adjustment through parallel adjustment across multiple segments, thus achieving a balance between accuracy and efficiency.

[0060] In one alternative implementation, the neural network is deployed in a many-core system, which includes multiple processing cores, and multiple network segments are deployed on multiple processing cores respectively.

[0061] In one optional implementation, the calibration data corresponding to the calibration dataset includes: multiple scene calibration data corresponding to multiple scene calibration datasets; wherein, the multiple scene calibration datasets correspond to multiple application scenarios; and the network weights of the neural network include: multiple scene network weights corresponding to multiple application scenarios.

[0062] To facilitate understanding, the following examples illustrate the implementation of the neural network-based weight quantization method provided in this disclosure:

[0063] Large models (LMs), especially large language models (LLMs) such as GPT-3 (Generative Pre-trained Transformer-3) and GLM (General Language Model), have massive parameters exceeding hundreds of billions, posing significant challenges to deployment and application. Simultaneously, with increasing demands for privacy protection and timeliness, the necessity for local deployment of LLMs on terminal devices such as mobile phones and laptops is growing. Therefore, higher requirements are placed on further reducing the storage space and bandwidth of LMs during computation. In inference scenarios, the computational cost of generating each token for large language models is far less than the transmission cost of weight parameters. Therefore, compressing the total storage or bandwidth of large language models can significantly improve inference speed.

[0064] Weight compression is an effective method to solve the above problems. Weight compression mainly involves model quantization, model pruning, and low-rank decomposition. For example, model quantization refers to reducing the precision of the magnitude representation of model weights by compressing the 16-bit floating-point (fp16) representation of weights into 8-bit integers (int8), 4-bit integers (int4), or even lower bit widths. Compared to the performance of fp16 models, int4 precision quantization does not result in a significant decrease. Model pruning, on the other hand, involves discarding unimportant weights in the model parameters to achieve sparsity. Low-rank decomposition utilizes the low-rank property of the weight matrix to discard some singular values ​​and singular vectors with less influence, decomposing a large weight matrix into a product of smaller weight matrices, thereby reducing parameter storage. Both model pruning and low-rank decomposition significantly reduce model precision, thus greatly impacting model accuracy. Therefore, model quantization (i.e., quantizing the weights in the neural network model) is usually adopted.

[0065] In related technologies, the model quantization process can be described by the following formula:

[0066]

[0067] Among them, W, These represent the weights before and after quantization, respectively, and can take various forms such as weight matrices or weight vectors. Here, s is the scaling factor, typically the maximum absolute value of W. n is the quantization bit width. W can be the original weights of the neural network mentioned above. The original weights mentioned above can be quantized weights corresponding to a preset truncation ratio (such as the first quantization weight or the second quantization weight). Correspondingly, the dequantization process can be expressed by the following formula:

[0068]

[0069] Among them, because there are a small number of outliers in the weights, the presence of these values ​​makes s larger, making it more difficult to distinguish the weight elements with concentrated values ​​(whose absolute values ​​are generally small), thus resulting in a larger quantization error. Figure 2 A schematic diagram of outliers in the weights is shown. Wherein, Figure 2 The horizontal axis represents the index of the weighted elements (sorted from largest to smallest, not corresponding to their original positions), and the vertical axis represents the element's value. A small region with a steep gradient on both sides is the outlier region. For example... Figure 2 As shown, after sorting the weight values ​​from largest to smallest, there are a few outliers.

[0070] Therefore, in related technologies, the outliers with the largest absolute values ​​in W are first saturated. That is, after taking the absolute values ​​of the weighted elements in W and sorting them, a certain proportion of the largest elements are saturated to smaller, larger values ​​(i.e., truncation is performed). The above process can be represented by the following code snippet:

[0071] This code uses the PyTorch library to process tensor data (i.e., weights), mainly completing the following two steps:

[0072] First, compute an upper bound on the absolute values ​​of a tensor (weights) using the following code, and then use that upper bound to clamp the values ​​in the original tensor:

[0073] `upperbound=torch.quantile(data.abs().float(),ratio,dim=-1,keepdim=True)`:

[0074] -`data.abs()` calculates the absolute value of each element in the tensor `data`.

[0075] The `.float()` function converts the tensor to a 32-bit floating-point format (`torch.float32`). This is to ensure that the `torch.quantile` function can execute correctly, as it may require a specific data type in some cases.

[0076] The `torch.quantile(...,ratio,dim=-1,keepdim=True)` function calculates the quantiles along the last dimension (`dim=-1`), where `ratio` is a value between 0 and 1 representing the desired quantile position. For example, if `ratio` is 0.95, it calculates the 95th quantile, meaning it retrieves the value at the 0.95th position after sorting in ascending order. `keepdim=True` ensures that the result preserves the dimension of the input tensor.

[0077] `upperbound` is the quantile of the maximum value of each slice along the last dimension, which can be used as a threshold for subsequent operations.

[0078] Next, execute the following code:

[0079] `data=torch.clamp(data,min=-upperbound,max=upperbound).to(torch.bfloat16)`:

[0080] `torch.clamp(data, min = -upperbound, max = upperbound)` applies a clamping operation, restricting all values ​​in the `data` tensor to between `-upperbound` and `upperbound`. This means that any value less than `-upperbound` will be set to `-upperbound`, and any value greater than `upperbound` will be set to `upperbound`.

[0081] The `.to(torch.bfloat16)` method converts the tensor's data type to `bfloat16` (Brain FloatingPoint), a mixed-precision format commonly used in machine learning applications because it offers a good numerical range and low memory footprint.

[0082] The code above can be used to prevent problems such as gradient explosion or feature scaling, and to preprocess or normalize data during neural network training. This allows control over the dynamic range of the data, making it more suitable for the model training process.

[0083] Therefore, in related technologies, the original weights of the neural network can be truncated based on the ratio value (a specific form of the truncation ratio mentioned above), and quantization can be performed based on the truncated weights, thereby avoiding the influence of outliers and improving the quantization effect. However, in the outlier truncation method described above, the ratio parameter (used to determine what proportion of elements with the largest absolute value are saturated) needs to be manually adjusted, which leads to high time consumption and difficulty in obtaining the optimal configuration.

[0084] To address the aforementioned issues, the following example aims to design a method for automatically determining the outlier cutoff ratio (i.e., the ratio parameter) in order to reduce the cost of manual parameter setting, improve the optimization of parameter settings, and reduce quantization errors.

[0085] Example 1

[0086] This example proposes a method for determining the ratio parameter for each weighted element, specifically by traversing a single weight or a weight row.

[0087] Figure 3 The specific flowchart of Example 1 is shown, such as Figure 3 As shown, it includes the following steps:

[0088] Step S301: Obtain the calibration data of the neural network corresponding to the calibration dataset.

[0089] This process involves obtaining a calibration dataset, consisting of multiple representative typical inputs that are expected to match the input distribution. The quality of the calibration dataset is crucial, as it determines the quality of the solution. On the calibration dataset, the model infers and obtains intermediate computation results, saving the input (or output) data for each operator to be quantized (e.g., the linear operator mentioned above). This yields the input for each operator to be quantized, i.e., the calibration input. The calibration input is the specific form of the calibration data mentioned above.

[0090] For example, for each operator to be quantized in the model (such as convolutional layers, fully connected layers, etc.), obtain its intermediate computation results (i.e., input data) on the calibration dataset. Specifically, during the model inference process, hooks or callbacks can be inserted to capture the input of each operator and save this input data as calibration input for subsequent quantization parameter optimization.

[0091] Step S302: Obtain the first quantization weight corresponding to the preset first truncation ratio of the original weight of the neural network, and determine the first quantization error of the first quantization weight relative to the original weight based on the calibration data.

[0092] The preset first cutoff ratio can be any cutoff ratio within a preset ratio range.

[0093] Step S303: Based on multiple cutoff ratios different from the first cutoff ratio contained in the preset ratio range, the first cutoff ratio is adjusted multiple times to obtain multiple adjusted first cutoff ratios.

[0094] For example, for each operator to be quantized, given a calibration input, the ratios within a certain range are traversed, and multiple adjusted first cutoff ratios are determined based on the traversal results. For instance, the preset ratio range (e.g., 0.9800 to 0.9999, with a step size of 0.0001, i.e., 200 parameter settings) can be traversed to calculate the corresponding error in the operator output before and after quantization.

[0095] Step S304: Obtain the second quantization weights corresponding to the original weights and the multiple adjusted first cutoff ratios, and obtain the multiple second quantization weights corresponding to the multiple adjusted first cutoff ratios.

[0096] Step S305: For any one of the multiple second quantization weights, determine the second quantization error of the second quantization weight relative to the original weight based on the calibration data, and obtain multiple second quantization errors corresponding to the multiple second quantization weights.

[0097] The formula for calculating the first quantization error or the second quantization error mentioned above can be the following formula:

[0098]

[0099] Where W is the high-precision weight matrix before quantization. It is the reconstructed weight matrix after outlier truncation quantization and dequantization under a certain ratio setting, where X is the calibration input.

[0100] Step S306: Based on the preset calculation results between multiple second quantization errors, determine the second quantization error that satisfies the preset convergence condition from among the multiple second quantization errors.

[0101] For example, after obtaining the error under different ratio settings, you can choose the ratio with the smallest error as the parameter setting for that weight.

[0102] Step S307: Based on the adjusted first truncation ratio corresponding to the second quantization error that satisfies the preset convergence condition, the original weights are truncated and quantized to obtain the network weights of the neural network.

[0103] Typically, a weight refers to a weight matrix, and the corresponding unit for quantization can be a row in the weight matrix (within the same output channel). Therefore, the ratio can be shared by a single weight, meaning all rows of that weight share the same ratio. Alternatively, a ratio can be set separately for each weight row, determined in a similar way to the above process, i.e., performing the above procedure for each weight row.

[0104] Specifically, the granularity of the ratio setting can match the granularity of the shared scale (i.e., the scaling factor s in the quantization formula). For example, if a weight matrix shares a scale, then a ratio is set for that weight matrix. If a weight row shares a scale, then a ratio is set for that weight row. If a quarter of a weight row shares a scale, then a ratio is set for that quarter of a weight row. In other words, the granularity of the truncation ratio setting mentioned above can be flexibly adjusted according to the actual scenario. Specifically, it can be: setting a truncation ratio (i.e., ratio) for the complete weight matrix corresponding to an operator, setting a truncation ratio for each row or column in the complete weight matrix corresponding to an operator, or further dividing each row or column in the complete weight matrix corresponding to an operator into multiple regions and setting a truncation ratio for each region. In short, this application does not limit the granularity of the truncation ratio adaptation.

[0105] Example 2

[0106] The parameter-fixing method in Example 1 mentioned above essentially involves iterating through a range of feasible solutions for a given objective function (i.e., the error function) to obtain the optimal solution. However, considering the high cost of this iteration, especially when a ratio needs to be determined for each weight row, a computational cost is significant. For example, assuming a typical input dimension of a 7B (billion) model is 4096 with approximately 200 weights, and each row requires 200 iterations, a total of 163,840,000 calculations are needed, resulting in substantial computational cost.

[0107] To improve computational efficiency, Example 2 can use a non-gradient optimization method to determine the ratio. In the process of implementing this invention, the inventors discovered that the objective function (error) is not differentiable with respect to the ratio, and therefore, a non-gradient optimization method is used. Non-gradient optimization refers to a method that does not rely on the derivative (gradient) information of the objective function for searching and optimization. It is suitable for situations where the objective function is not differentiable, has noise, or its gradient is difficult to calculate. Compared with gradient-based methods (such as gradient descent), non-gradient optimization algorithms are generally more robust and can find solutions to a wider range of optimization problems. Non-gradient optimization methods generally have the following characteristics: (1) Derivative-independent: Non-gradient methods do not require gradient information of the objective function, which allows them to work without explicit gradient information. (2) Global optimization: Many non-gradient methods are designed to find global optima rather than local optima. This example can specifically use the following non-gradient optimization methods: (1) Direct search method: For example, the Nelder-Mead simplex method (also known as the Downhill Simplex method) searches for the minimum by iteratively updating the vertices of a polyhedron. (2) Evolutionary algorithms: such as Genetic Algorithms (GA) and Differential Evolution (DE), which are inspired by natural selection and biological evolution and search for optimal solutions by simulating the process of natural selection. (3) Particle Swarm Optimization (PSO): a swarm intelligence-based optimization method where each particle represents a candidate solution and adjusts its flight direction based on its historical best position and the best position of the swarm. (4) Simulated Annealing (SA): inspired by the annealing phenomenon during metal cooling, this method allows for the acceptance of inferior solutions with a certain probability, thus helping to escape local optima. (5) Ant Colony Optimization (ACO): an optimization method inspired by the foraging behavior of ants, which find the shortest path from their nest to a food source by releasing and tracking pheromones.

[0108] In Example 2, the implementation is mainly based on the zeroth-order gradient optimization method:

[0109] Specifically, firstly, for each weight or weight row, set an initial ratio (i.e., the preset first cutoff ratio), which can be a random initialization result within a certain range, such as randomly selecting a number with equal probability from 0.9800 to 0.9999 as the initial value.

[0110] Then, a random perturbation δ (i.e., the i-th random perturbation value mentioned above) is obtained. This random perturbation can be a Gaussian or uniform distribution perturbation within a certain range (here, it is one-dimensional, i.e., a single number). Next, the error is calculated under the outlier cutoff ratios ratio+δ (i.e., the first sub-ratio corresponding to the first operation) and ratio-δ (i.e., the second sub-ratio corresponding to the second operation), respectively. The ratio is then updated to the ratio value with the smaller error value. For example, if the error corresponding to ratio+δ is smaller, then the ratio can be updated to ratio+δ. This process is repeated multiple times until convergence. One convergence condition is that the values ​​of the two errors are sufficiently close.

[0111] Finally, the final ratio is set as the outlier truncation ratio to obtain the network weights of the neural network based on this outlier truncation ratio.

[0112] In addition, other non-gradient optimization methods such as the golden section method are also feasible.

[0113] Example 3

[0114] Example 3 aims to achieve non-gradient optimization between multiple weights or weight rows. Specifically, while Example 2 avoids traversal, it still requires solving for each weight or weight row, resulting in a large number of calculations. In other words, Example 2 actually performs independent calculations for the weights of multiple linear operators in the neural network. Therefore, Example 2 does not consider the mutual influence between the weights to be quantized corresponding to each linear operator during quantization, which can easily lead to problems such as error accumulation. For example, if the quantization result of a certain operator in Example 2 is poor, the resulting error may further propagate to adjacent operators that have an input-output relationship with that operator, causing error accumulation and resulting in a large deviation in the final quantization result. To solve the above problems, in the example, all weights in a neural network model can be co-optimized. The algorithm flow is as follows:

[0115] Obtain the initial ratios of all weights or weight rows of a model (using the same method as above), and concatenate the ratios into a vector, which is the variable to be solved. Update the ratio vector using a non-gradient optimization method until convergence, using the same method as above.

[0116] The initial ratio of all weights or weight rows of a model can be applied to the entire neural network mentioned above, or to any network segment in the neural network model.

[0117] Furthermore, since this involves multiple weights or weight rows within a neural network model or a network segment, the error at this point should be the average (or maximum, without restriction, of the errors of each weight or weight row). In addition, because the ratios of all weights in the model are co-optimized at this stage, the error can be set as the final output reconstruction error of the entire model. One example is...

[0118]

[0119] Where X represents the model input, f represents the model, and θ represents the parameters before quantization. This represents the reconstructed parameters after outlier truncation quantization and dequantization under a certain ratio setting (where ratio is a vector).

[0120] It should be noted that collaborative optimization can also be performed on a subset of weights or weight rows, rather than the entire model, which can improve the parallelism of the solution. For example, the weights of a model can be divided into multiple groups, and collaborative optimization can be performed within each group. These groups can be partitioned using the network segmentation method mentioned above. The definition of error can refer to the entire model; it can be the average error of each weight, or the error can be calculated using the input and output of modules composed of multiple weight groups.

[0121] Example 4

[0122] Example 4 can add scenario features based on Examples 1, 2, or 3. The quantization error assessment is based on the input calibration data. The previous examples were implemented by obtaining a quantization model from a single set of calibration data. Example 4 can be implemented by preparing multiple sets of calibration data suitable for different fields (or application scenarios, such as medicine, programming, etc.), and obtaining multiple quantization models respectively according to the methods in Examples 1, 2, or 3. This allows for the preparation of the most suitable quantization model for different fields, resulting in better accuracy than a single quantization model adapted to a general scenario.

[0123] It is important to note that obtaining multiple quantization models can involve pre-quantizing and saving the quantized weights. This results in more models and larger storage requirements. However, when deploying applications, quantization models can still reduce the amount of data transferred and computationally required to the computing device, thus achieving energy efficiency advantages.

[0124] Another approach is to save only one set of high-precision weights (e.g., fp16), obtain a set of calibration data based on the scenario when deploying the application, complete the quantization, and then deploy the application. In this way, only one set of high-precision weights needs to be saved, and quantization can be performed in real time when the application is deployed.

[0125] Example 5

[0126] This example focuses on how to accelerate the automatic outlier truncation quantization process on many-core chips. It's important to note that the automatic parameter determination method in Example 5 is highly parallelized. The granularity of parameter determination can be a single weight, a row of weights, or a group of weights, and these do not affect each other. Therefore, different groups can be deployed on various cores of the many-core chip to complete the automatic parameter determination process in parallel. One implementation is as follows: First, the large model is deployed on the many-core chip, with one core or a group of cores handling the calculation of one weight or a group of weights. Then, the model performs inference on the calibration dataset and saves the intermediate results of the inference, i.e., the calibration inputs of the operators to be quantized, to the corresponding core. Finally, the parameter determination algorithm is executed in parallel for each core or group of cores to determine the optimal parameters and complete the parameter quantization.

[0127] Because the computation during quantization is performed locally on multiple processing cores, and the computation processes on these cores are independent of each other, significant data transfer between cores is avoided, reducing power consumption and improving parallelism. Furthermore, error calculations can be dynamically allocated to additional processing cores (such as other idle cores) to further reduce the overhead of the cores actually performing the computational tasks. In this example, multiple network segments can be deployed across multiple processing cores to enhance parallelism.

[0128] The examples above can be used individually or in combination.

[0129] Additionally, it should be noted that the neural network in the above example can be used to process text data and / or image data, specifically in fields such as text data or visual processing.

[0130] In summary, the above method can automatically determine the truncation ratio of the outlier truncation quantization method. Through the collaborative optimization of multiple weight quantizations of the entire model, it can reduce the complexity of parameter setting, reduce quantization error, and improve the accuracy of the model.

[0131] Through the examples above, we can demonstrate how to determine the outlier truncation ratio through various methods: traversal (e.g., determining it based on the granularity of the ratio setting), non-gradient optimization, collaborative parameter-determining quantization via multi-weight grouping, and domain-adaptive quantization. Furthermore, the operators mentioned can be linear (matrix multiplication by vector), convolution, etc. The neural network models described can take various forms, including large language models, large visual models, and multimodal models.

[0132] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0133] Furthermore, another embodiment of this disclosure provides a many-core chip, which can be used to run a neural network comprising multiple network segments. The neural network includes multiple network segments, and the many-core chip includes multiple processing cores corresponding to the multiple network segments. Each processing core is used to run a corresponding network segment in the neural network. Furthermore, each processing core performs weight quantization processing on the weights of multiple target linear operators corresponding to the target linear operators in the corresponding network segment based on the above method. The network segments can be divided according to network layers or network functional modules in the neural network, specifically, a network layer or a network functional module. Alternatively, a network layer or a network functional module can be further divided into multiple network segments, each network segment containing one or more operators.

[0134] Figure 4 This is a block diagram of a neural network-based weight quantization device provided in an embodiment of this disclosure. (Refer to...) Figure 4 This disclosure provides a weight quantization device based on a neural network, the device comprising:

[0135] The acquisition module 41 is adapted to acquire the calibration data of the neural network corresponding to the calibration dataset;

[0136] The determining module 42 is adapted to obtain the first quantized weight corresponding to the original weight of the neural network at a preset first truncation ratio, and to determine the first quantization error of the first quantized weight relative to the original weight based on the calibration data;

[0137] The processing module 43 is adapted to perform truncation and quantization processing on the original weights of the neural network according to the first truncation ratio when it is determined from the calibration data that the first quantization error meets the preset convergence condition, so as to obtain the network weights of the neural network.

[0138] In one optional implementation, the original weights of the neural network include at least one of the following: a weight matrix corresponding to a preset operator in the neural network, at least one matrix unit in the weight matrix corresponding to the preset operator in the neural network, and at least one matrix subunit in the at least one matrix unit;

[0139] The matrix unit includes: matrix rows and / or matrix columns, and the matrix subunit includes: a combination of multiple row elements contained in a matrix row and / or a combination of multiple column elements contained in a matrix column;

[0140] Furthermore, the preset first truncation ratio includes at least one of the following: a first matrix truncation ratio corresponding to the weight matrix of the network operator in the neural network, a first unit truncation ratio corresponding to at least one matrix unit in the weight matrix, and a first sub-unit truncation ratio corresponding to the at least one matrix sub-unit.

[0141] In an optional implementation, the determining module is further configured to: when it is determined from the calibration data that the first quantization error does not meet the preset convergence condition, adjust the first cutoff ratio according to the first quantization error to obtain the adjusted first cutoff ratio; obtain the second quantization weight corresponding to the adjusted first cutoff ratio of the original weight, and determine the second quantization error of the second quantization weight relative to the original weight according to the calibration data;

[0142] The processing module is used to truncate and quantize the original weights of the neural network according to the adjusted first truncation ratio, provided that the second quantization error meets the preset convergence condition based on the calibration data.

[0143] In one optional implementation, the preset first cutoff ratio is any cutoff ratio within a preset ratio range;

[0144] The determining module is specifically used for:

[0145] Based on the first quantization error and the multiple cutoff ratios contained in the preset ratio range, the first cutoff ratio is adjusted multiple times to obtain multiple adjusted first cutoff ratios.

[0146] In one optional implementation, the preset ratio range includes: a first interval endpoint and a second interval endpoint, and the preset first cutoff ratio is the cutoff ratio located at the first interval endpoint in the preset ratio range;

[0147] The determining module is specifically used to: traverse multiple cutoff ratios in the preset ratio interval along the direction from the endpoint of the first interval to the endpoint of the second interval, and sequentially determine each cutoff ratio obtained by the traversal as the adjusted first cutoff ratio; and the second quantization error includes: multiple second quantization errors corresponding to multiple second quantization weights, wherein the multiple second quantization weights correspond one-to-one with the multiple adjusted first cutoff ratios;

[0148] The second quantization error that satisfies the preset convergence condition is the second quantization error with the smallest error value among the plurality of second quantization errors.

[0149] In one alternative implementation, the determining module is specifically used for:

[0150] Generate the i-th random perturbation value, and adjust the first cutoff ratio for the i-th time according to the i-th random perturbation value to obtain the i-th set of adjusted first cutoff ratios corresponding to the i-th random perturbation value;

[0151] The second quantization error includes: the second quantization error of the i-th group of second quantization weights determined according to the calibration data relative to the original weights; wherein the i-th group of second quantization weights is the quantization weight of the original weights corresponding to the i-th group of adjusted first cutoff ratios;

[0152] The processing module is specifically used for:

[0153] Calculate the change in the second quantization error of the i-th group relative to the second quantization error of the (i-1)-th group. If the change in the error is less than a preset change threshold, determine that the second quantization error of the i-th group satisfies the preset convergence condition; where i is a natural number.

[0154] In one optional implementation, the i-th group of adjusted first truncation ratios includes: a first sub-ratio corresponding to the first operation and a second sub-ratio corresponding to the second operation;

[0155] The determining module is specifically used for:

[0156] The first operation is performed on the first truncation ratio based on the i-th random perturbation value to obtain the first sub-ratio corresponding to the first operation; and the second operation is performed on the first truncation ratio based on the i-th random perturbation value to obtain the second sub-ratio corresponding to the second operation.

[0157] Then the second quantization error of the i-th group is the minimum value between the first sub-error corresponding to the first sub-proportion and the second sub-error corresponding to the second sub-proportion;

[0158] Furthermore, if it is determined that the second quantization error of the i-th group does not meet the preset convergence condition, the (i+1)-th random perturbation value is generated, and the first truncation ratio is adjusted for the (i+1)-th time according to the (i+1)-th random perturbation value to obtain the (i+1)-th adjusted first truncation ratio corresponding to the (i+1)-th random perturbation value.

[0159] In one optional implementation, the preset operator in the neural network is a linear operator, and the calibration data of the neural network corresponding to the calibration dataset is the calibration input data and / or calibration output data corresponding to the linear operator;

[0160] Wherein, when the preset operator in the neural network includes multiple target linear operators of the neural network, the calibration data of the neural network corresponding to the calibration dataset is the calibration input data corresponding to the target linear operator used as the network input end among the multiple target linear operators, and / or the calibration output data corresponding to the target linear operator used as the network output end;

[0161] Furthermore, the preset first truncation ratio is represented by a first truncation ratio vector, which includes: multiple first vector elements corresponding to the multiple target linear operators, each first vector element being used to truncate the original weights of the corresponding target linear operator in the neural network;

[0162] Furthermore, the original weights of the neural network include: multiple original weights corresponding to multiple target linear operators in the neural network; the network weights of the neural network include: multiple operator weights corresponding to multiple target linear operators in the neural network;

[0163] The first quantization weight includes: a plurality of first quantization weights corresponding to the plurality of original weights, wherein any first quantization weight is determined by: for any original weight corresponding to any target linear operator in the neural network, selecting a first vector element corresponding to the target linear operator from a plurality of first vector elements of the first truncation ratio vector, and obtaining the first quantization weight corresponding to the target linear operator based on the selected first vector element.

[0164] In one optional implementation, the neural network comprises multiple network segments, and the multiple target linear operators in the neural network correspond to any one of the network segments;

[0165] Furthermore, the first truncation ratio vector includes: multiple vector elements corresponding to multiple target linear operators in any network segment, each vector element being used to truncate the original weights of the target linear operators corresponding to any network segment; then the network weights of the neural network include: multiple operator weights corresponding to multiple target linear operators in any network segment.

[0166] In one optional implementation, the calibration data corresponding to the calibration dataset includes: multiple scene calibration data corresponding to multiple scene calibration datasets; wherein, the multiple scene calibration datasets correspond to multiple application scenarios;

[0167] Furthermore, the network weights of the neural network include: multiple scenario network weights corresponding to multiple application scenarios.

[0168] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0169] Reference Figure 5 This disclosure provides an electronic device, which includes: at least one processor 901; at least one memory 902; and one or more I / O interfaces 903 connected between the processor 901 and the memory 902; wherein the memory 902 stores one or more computer programs that can be executed by the at least one processor 901, and the one or more computer programs are executed by the at least one processor 901 to enable the at least one processor 901 to execute the above-described neural network-based weight quantization method.

[0170] In some embodiments, the processing device can be a neuromorphic chip. Since neuromorphic chips can employ vectorized computation and require external memory, such as Double Data Rate (DDR) synchronous dynamic random access memory, to load parameters like weights of the neural network model, the batch processing method used in this disclosure provides higher computational efficiency.

[0171] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the aforementioned neural network-based weight quantization method. The computer-readable storage medium may be volatile or non-volatile.

[0172] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0173] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0174] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0175] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0176] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0177] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0178] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0179] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0180] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0182] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A weight quantization method based on a neural network, the method comprising: Obtain the calibration data corresponding to the calibration dataset for the neural network; Obtain the original weights of the neural network corresponding to a first quantized weight with a preset first truncation ratio, and determine the first quantization error of the first quantized weight relative to the original weight based on the calibration data; If the calibration data determines that the first quantization error meets the preset convergence condition, the original weights of the neural network are truncated and quantized according to the first truncation ratio to obtain the network weights of the neural network.

2. The method according to claim 1, wherein, The original weights of the neural network include at least one of the following: a weight matrix corresponding to a preset operator in the neural network, at least one matrix unit in the weight matrix corresponding to the preset operator in the neural network, and at least one matrix subunit in the at least one matrix unit; The matrix unit includes: matrix rows and / or matrix columns, and the matrix subunit includes: a combination of multiple row elements contained in a matrix row and / or a combination of multiple column elements contained in a matrix column; Furthermore, the preset first truncation ratio includes at least one of the following: a first matrix truncation ratio corresponding to the weight matrix of the network operator in the neural network, a first unit truncation ratio corresponding to at least one matrix unit in the weight matrix, and a first sub-unit truncation ratio corresponding to the at least one matrix sub-unit.

3. The method according to claim 2, wherein, After determining the first quantization error of the first quantization weight relative to the original weight based on the calibration data, the method further includes: If it is determined from the calibration data that the first quantization error does not meet the preset convergence condition, the first cutoff ratio is adjusted according to the first quantization error to obtain the adjusted first cutoff ratio. Obtain the second quantization weight corresponding to the adjusted first cutoff ratio of the original weight, and determine the second quantization error of the second quantization weight relative to the original weight based on the calibration data; The step of truncating and quantizing the original weights of the neural network according to the first truncation ratio when the first quantization error is determined to meet the preset convergence condition based on the calibration data includes: truncating and quantizing the original weights of the neural network according to the adjusted first truncation ratio when the second quantization error is determined to meet the preset convergence condition based on the calibration data.

4. The method according to claim 3, wherein, The preset first cutoff ratio is any cutoff ratio within a preset ratio range; The step of adjusting the first truncation ratio based on the first quantization error includes: Based on the first quantization error and the multiple cutoff ratios contained in the preset ratio range, the first cutoff ratio is adjusted multiple times to obtain multiple adjusted first cutoff ratios.

5. The method according to claim 4, wherein, The preset ratio range includes: the endpoint of the first range and the endpoint of the second range, and the preset first cutoff ratio is the cutoff ratio located at the endpoint of the first range in the preset ratio range; The step of adjusting the first truncation ratio multiple times based on multiple truncation ratios included in the preset ratio range to obtain the multiple adjusted first truncation ratios includes: Along the direction from the endpoint of the first interval to the endpoint of the second interval, traverse multiple cutoff ratios in the preset ratio interval, and sequentially determine each cutoff ratio obtained by traversal as the adjusted first cutoff ratio; and the second quantization error includes: multiple second quantization errors corresponding to multiple second quantization weights, wherein the multiple second quantization weights correspond one-to-one with the multiple adjusted first cutoff ratios; The second quantization error that satisfies the preset convergence condition is the second quantization error with the smallest error value among the plurality of second quantization errors.

6. The method according to claim 3, wherein, The step of adjusting the first cutoff ratio based on the first quantization error to obtain the adjusted first cutoff ratio includes: Generate the i-th random perturbation value, and adjust the first cutoff ratio for the i-th time according to the i-th random perturbation value to obtain the i-th set of adjusted first cutoff ratios corresponding to the i-th random perturbation value; The second quantization error includes: the second quantization error of the i-th group of second quantization weights determined according to the calibration data relative to the original weights; wherein the i-th group of second quantization weights is the quantization weight of the original weights corresponding to the i-th group of adjusted first cutoff ratios; The step of determining that the second quantization error satisfies the preset convergence condition based on the calibration data includes: Calculate the change in the second quantization error of the i-th group relative to the second quantization error of the (i-1)-th group. If the change in the error is less than a preset change threshold, determine that the second quantization error of the i-th group satisfies the preset convergence condition; where i is a natural number.

7. The method according to claim 6, wherein, The first truncation ratio after adjustment of the i-th group includes: a first sub-ratio corresponding to the first operation, and a second sub-ratio corresponding to the second operation; The step of adjusting the first cutoff ratio for the i-th time based on the i-th random perturbation value to obtain the i-th adjusted first cutoff ratio corresponding to the i-th random perturbation value includes: The first operation is performed on the first truncation ratio based on the i-th random perturbation value to obtain the first sub-ratio corresponding to the first operation; and the second operation is performed on the first truncation ratio based on the i-th random perturbation value to obtain the second sub-ratio corresponding to the second operation. Then the second quantization error of the i-th group is the minimum value between the first sub-error corresponding to the first sub-proportion and the second sub-error corresponding to the second sub-proportion; Furthermore, if it is determined that the second quantization error of the i-th group does not meet the preset convergence condition, the (i+1)-th random perturbation value is generated, and the first truncation ratio is adjusted for the (i+1)-th time according to the (i+1)-th random perturbation value to obtain the (i+1)-th adjusted first truncation ratio corresponding to the (i+1)-th random perturbation value.

8. The method according to any one of claims 2-7, wherein, The preset operator in the neural network is a linear operator, and the calibration data of the neural network corresponding to the calibration dataset is the calibration input data and / or calibration output data corresponding to the linear operator; Wherein, when the preset operator in the neural network includes multiple target linear operators of the neural network, the calibration data of the neural network corresponding to the calibration dataset is the calibration input data corresponding to the target linear operator used as the network input end among the multiple target linear operators, and / or the calibration output data corresponding to the target linear operator used as the network output end; Furthermore, the preset first truncation ratio is represented by a first truncation ratio vector, which includes: multiple first vector elements corresponding to the multiple target linear operators, each first vector element being used to truncate the original weights of the corresponding target linear operator in the neural network; Furthermore, the original weights of the neural network include: multiple original weights corresponding to multiple target linear operators in the neural network; the network weights of the neural network include: multiple operator weights corresponding to multiple target linear operators in the neural network; The first quantization weight includes: a plurality of first quantization weights corresponding to the plurality of original weights, wherein any first quantization weight is determined by: for any original weight corresponding to any target linear operator in the neural network, selecting a first vector element corresponding to the target linear operator from a plurality of first vector elements of the first truncation ratio vector, and obtaining the first quantization weight corresponding to the target linear operator based on the selected first vector element.

9. The method according to claim 8, wherein, The neural network comprises multiple network segments, and the multiple target linear operators in the neural network correspond to any one of the network segments; Furthermore, the first truncation ratio vector includes: multiple vector elements corresponding to multiple target linear operators in any network segment, each vector element being used to truncate the original weights of the target linear operators corresponding to any network segment; then the network weights of the neural network include: multiple operator weights corresponding to multiple target linear operators in any network segment.

10. The method according to any one of claims 1-6, wherein, The calibration data corresponding to the calibration dataset includes: multiple scenario calibration data corresponding to multiple scenario calibration datasets; wherein, the multiple scenario calibration datasets correspond to multiple application scenarios; Furthermore, the network weights of the neural network include: multiple scenario network weights corresponding to multiple application scenarios.

11. A many-core chip, said many-core chip being used to run a neural network comprising multiple network segments, wherein, The neural network includes multiple network segments, and the many-core chip includes multiple processing cores corresponding to the multiple network segments; In this process, any one of the processing kernels is used to run the corresponding network segment in the neural network; and, the processing kernel performs weight quantization processing on the weights of multiple operator weights corresponding to multiple target linear operators in the corresponding network segment based on the method described in any one of claims 1-10.

12. A weight quantization device based on a neural network, the device comprising: The acquisition module is adapted to acquire calibration data of the neural network corresponding to the calibration dataset; The determination module obtains the first quantized weight corresponding to the preset first truncation ratio of the original weights of the neural network, and determines the first quantization error of the first quantized weight relative to the original weights based on the calibration data; The processing module is adapted to, when it is determined from the calibration data that the first quantization error meets a preset convergence condition, truncate and quantize the original weights of the neural network according to the first truncation ratio to obtain the network weights of the neural network.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images