Quantization method, device and system of neural network model, electronic device and storage medium

Abstract

The disclosed embodiments disclose a quantization method, device and system, electronic device and medium of a neural network model, wherein the method comprises: obtaining the weight matrix of any network layer to be quantized in the neural network model; performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; quantizing the weight matrix to be quantized of any network layer to be quantized to obtain the quantized weight matrix of any network layer to be quantized; and obtaining the quantized neural network model based on the quantized weight matrix of any network layer to be quantized in the neural network model. The disclosed embodiments can reduce the distribution difference of weight data of each channel in the weight matrix, can reduce quantization error, and help improve the accuracy of the quantized neural network.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology, and in particular to a quantization method, device and system, electronic device and storage medium for a neural network model. Background Art

[0002] With the rapid development of artificial intelligence, artificial neural networks are being used more and more widely. The main operation in artificial neural networks is matrix-vector multiplication, such as convolutional layers and fully connected layers. The storage-computing neural network accelerator integrates the computing unit into the storage unit, which can efficiently run matrix-vector multiplication, thereby greatly reducing the frequent data interaction between the computing unit and the storage unit, and also greatly reducing the data interaction between the intermediate data and the off-chip main memory. Therefore, the use of storage-computing technology is a very promising development direction for neural network accelerators in the future.

[0003] The storage and computing unit array (crossbar) used in the storage and computing integrated structure has a fixed size. For example, a storage and computing unit array of size 128×128 can support the multiplication of up to 128 inputs and a 128×128 matrix. A matrix larger than the size of the storage and computing unit array needs to be mapped to multiple storage and computing unit arrays, and each storage and computing unit array performs part of the matrix-vector multiplication operation. Since almost all neural network accelerators use low-bit weights and activation values ​​for calculations, neural networks need to be quantized. The weights and activation values ​​of the neural network are converted from floating point numbers to lower-bit fixed-point numbers through a quantization algorithm. The lower the number of bits after quantization, the lower the energy cost of running the neural network and the storage cost of storing the neural network, but it will also reduce the prediction performance of the neural network.

[0004] In the process of implementing the present disclosure, the inventors found that in the existing quantization algorithm that uses the layer of a neural network as the granularity for quantization, the weights of each layer of the neural network share a set of quantization parameters. Since the distribution of weights of different channels in the same layer of the neural network is generally very different, using a set of quantization parameters will cause channels with a smaller distribution range to have a larger quantization error. If a lower number of quantization bits is used, the prediction performance of the neural network will be greatly lost; in the existing quantization algorithm that uses the channel of the neural network as the granularity for quantization, the weights of an output channel of each layer of the neural network share a set of quantization parameters. Since the quantization parameters of each channel are different, fine-grained operation of different channels will increase the design overhead of the hardware and reduce the operating efficiency of the hardware; in the existing quantization algorithm that uses the storage and computing unit array as the granularity for quantization, each layer of the neural network is mapped to multiple storage and computing unit arrays, and the weights of each storage and computing unit array share a set of quantization parameters. Since each storage and computing unit array stores the weights of different channels, their distribution will also be very different, and using a set of quantization parameters will also result in quantization errors. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a quantization method, device and system, electronic device and storage medium for a neural network model.

[0006] One aspect of an embodiment of the present disclosure provides a quantization method for a neural network model, including: obtaining a weight matrix of any network layer to be quantized in the neural network model; performing matrix transformation on the weight matrix of any network layer to be quantized to obtain a weight matrix to be quantized of any network layer to be quantized; quantizing the weight matrix to be quantized of any network layer to be quantized to obtain a quantized weight matrix of any network layer to be quantized; and obtaining a quantized neural network model based on the quantized weight matrix of any network layer to be quantized in the neural network model.

[0007] Optionally, in any of the above method embodiments of the present disclosure, quantizing the weight matrix to be quantized of any network layer to be quantized to obtain the quantized weight matrix of any network layer to be quantized includes: dividing the weight matrix to be quantized of any network layer to be quantized according to the size of the storage and computing unit array in the storage and computing integrated accelerator to obtain multiple sub-weight matrices to be quantized; quantizing any sub-weight matrix to be quantized in the multiple sub-weight matrices to be quantized respectively to obtain multiple quantized sub-weight matrices; mapping the multiple quantized sub-weight matrices to multiple storage and computing unit arrays in the storage and computing integrated accelerator respectively, and storing the multiple quantized sub-weight matrices by the multiple storage and computing unit arrays.

[0008] Optionally, in any of the above method embodiments of the present disclosure, after obtaining the quantized neural network model based on the quantized weight matrix of any network layer to be quantized in the neural network model to be quantized, it also includes: obtaining multiple calibration samples from a calibration sample set, providing the multiple calibration samples as input to the quantized neural network model respectively, and processing the input multiple calibration samples via the quantized neural network model to obtain the output of the quantized neural network model; adjusting the quantized weight matrix of any network layer to be quantized according to the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized; wherein the output of the neural network model to be quantized is the output obtained by providing the multiple calibration samples as input to the neural network model to be quantized, and processing the input multiple calibration samples via the neural network model to be quantized.

[0009] Optionally, in any of the above method embodiments of the present disclosure, after performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized, it also includes: initializing the number of matrix transformations to a preset value; randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange to obtain a new weight matrix to be quantized of any network layer to be quantized; quantizing the new weight matrix to be quantized of any network layer to be quantized to obtain a new quantized weight matrix of any network layer to be quantized, and obtaining a new quantized neural network model based on the new quantized weight matrix of any network layer to be quantized; determining the relationship between the output of the new quantized neural network model and the output of the neural network model to be quantized. cosine distance, and determine whether the cosine distance is less than a preset cosine distance, and determine whether the number of matrix transformations is less than a preset threshold; if the cosine distance is less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, determine the target probability based on the cosine distance and the preset cosine distance, and determine whether the target probability is greater than the preset probability; if the target probability is greater than the preset probability, use the cosine distance as the preset cosine distance, add 1 to the number of matrix transformations, and perform the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange; if the target probability is not greater than the preset probability, use the new weight matrix to be quantized of any network layer to be quantized as the weight matrix to be quantized of any network layer to be quantized.

[0010] Optionally, in any of the above method embodiments of the present disclosure, it also includes: if the cosine distance is less than the preset cosine distance and the number of matrix transformations is not less than the preset threshold, the new weight matrix to be quantized of any network layer to be quantized is used as the weight matrix to be quantized of any network layer to be quantized; if the cosine distance is not less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, the cosine distance is used as the preset cosine distance, the number of matrix exchanges is increased by 1, and the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed.

[0011] Optionally, in any of the above method embodiments of the present disclosure, the weight matrix of any network layer to be quantized is subjected to matrix transformation to obtain the weight matrix to be quantified of any network layer to be quantized, including: determining a joint network layer of any network layer to be quantized, wherein the joint network layer is any other network layer to be quantized in the neural network model to be quantized except the any network layer to be quantized, and the joint network layer takes the output of the any network layer to be quantized as input; determining a first transformation matrix of the weight matrix of any network layer to be quantized, and using the first transformation matrix to perform a matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; using the transposed matrix of the first transformation matrix as the second transformation matrix of the weight matrix of the joint network layer, and using the second transformation matrix to perform a matrix transformation on the weight matrix of the joint network layer to obtain the weight matrix to be quantized of the joint network layer.

[0012] Optionally, in any of the above method embodiments of the present disclosure, the first transformation matrix of the weight matrix of any network layer to be quantized includes: based on the corresponding relationship between the input channel of the joint network layer and the output channel of any network layer to be quantized, determining the corresponding relationship between the weight data of each output channel in the weight matrix of any network layer to be quantized and the weight data of each input channel in the weight matrix of the joint network layer; determining the distribution characteristic values ​​of the weight data of each output channel and the distribution characteristic values ​​of the weight data of each input channel, and multiplying the distribution characteristic values ​​of the weight data of each output channel with the distribution characteristic values ​​of the corresponding weight data of each input channel respectively to obtain the target characteristic values ​​of the weight data of each output channel; according to the target characteristic values ​​of the weight data of each output channel, the position of the weight data of each output channel in the weight matrix of any network layer to be quantized is transformed in order from small to large to obtain the weight matrix to be quantized of any network layer to be quantized, and the transformation matrix of the weight matrix to be quantized of any network layer to be quantized relative to the weight matrix of any network layer to be quantized is used as the first transformation matrix.

[0013] Another aspect of the disclosed embodiment provides a quantization device for a neural network model, including: an acquisition module, used to acquire the weight matrix of any network layer to be quantized in the neural network model to be quantized; a matrix transformation module, used to perform matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; a quantization module, used to quantize the weight matrix to be quantized of any network layer to be quantized to obtain the quantized weight matrix of any network layer to be quantized; a first acquisition module, used to obtain a quantized neural network model based on the quantized weight matrix of any network layer to be quantized in the neural network model to be quantized.

[0014] Optionally, in any of the above-mentioned device embodiments of the present disclosure, the quantization module includes: a division unit, which is used to divide the weight matrix to be quantized of any network layer to be quantized according to the size of the storage and computing unit array in the storage and computing integrated accelerator, so as to obtain multiple sub-weight matrices to be quantized; a quantization unit, which is used to quantize any sub-weight matrix to be quantized in the multiple sub-weight matrices to be quantized, so as to obtain multiple quantized sub-weight matrices; a mapping unit, which is used to map the multiple quantized sub-weight matrices to the multiple storage and computing unit arrays in the storage and computing integrated accelerator, so that the multiple quantized sub-weight matrices are stored by the multiple storage and computing unit arrays.

[0015] Optionally, in any of the above-mentioned device embodiments of the present disclosure, it also includes: a second obtaining module, used to obtain multiple calibration samples from a calibration sample set, provide the multiple calibration samples as input to the quantized neural network model respectively, and process the input multiple calibration samples via the quantized neural network model to obtain the output of the quantized neural network model; an adjustment module, used to adjust the weight matrix to be quantized of any network layer to be quantized according to the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized; wherein the output of the neural network model to be quantized is the output obtained by providing the multiple calibration samples as input to the neural network model to be quantized, and processing the input multiple calibration samples via the neural network model to be quantized.

[0016] Optionally, in any of the above-mentioned device embodiments of the present disclosure, the matrix transformation module is also used to: initialize the number of matrix transformations to a preset value; randomly select any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange, to obtain a new weight matrix to be quantized of any network layer to be quantized; quantize the new weight matrix to be quantized of any network layer to be quantized, to obtain a new quantized weight matrix of any network layer to be quantized, and based on the new quantized weight matrix of any network layer to be quantized, obtain a new quantized neural network model; determine the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized, and determine whether the cosine distance is less than a preset value. Assume a cosine distance, and determine whether the number of matrix transformations is less than a preset threshold; if the cosine distance is less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, determine the target probability based on the cosine distance and the preset cosine distance, and determine whether the target probability is greater than the preset probability; if the target probability is greater than the preset probability, use the cosine distance as the preset cosine distance, add 1 to the number of matrix transformations, and perform the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange; if the target probability is not greater than the preset probability, use the new weight matrix to be quantized of any network layer to be quantized as the weight matrix to be quantized of any network layer to be quantized.

[0017] Optionally, in any of the above-mentioned device embodiments of the present disclosure, the matrix transformation module is also used to: if the cosine distance is less than the preset cosine distance and the number of matrix transformations is not less than the preset threshold, use the new weight matrix to be quantized of any network layer to be quantized as the weight matrix to be quantized of any network layer to be quantized; if the cosine distance is not less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, use the cosine distance as the preset cosine distance, increase the number of matrix exchanges by 1, and perform the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange.

[0018] Optionally, in any of the above-mentioned device embodiments of the present disclosure, the matrix transformation module includes: a determination unit, used to determine the joint network layer of any network layer to be quantized, the joint network layer is any other network layer to be quantized in the neural network model to be quantized except any network layer to be quantized, and the joint network layer takes the output of any network layer to be quantized as input; a first transformation unit, used to determine the first transformation matrix of the weight matrix of any network layer to be quantized, and use the first transformation matrix to perform a matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; a second transformation unit, used to use the transposed matrix of the first transformation matrix as the second transformation matrix of the weight matrix of the joint network layer, and use the second transformation matrix to perform a matrix transformation on the weight matrix of the joint network layer to obtain the weight matrix to be quantized of the joint network layer.

[0019] Optionally, in any of the above-mentioned device embodiments of the present disclosure, the first transformation matrix of determining the weight matrix of any network layer to be quantized includes: based on the corresponding relationship between the input channel of the joint network layer and the output channel of any network layer to be quantized, determining the corresponding relationship between the weight data of each output channel in the weight matrix of any network layer to be quantized and the weight data of each input channel in the weight matrix of the joint network layer; determining the distribution characteristic value of the weight data of each output channel and the distribution characteristic value of the weight data of each input channel, and multiplying the distribution characteristic value of the weight data of each output channel with the distribution characteristic value of the corresponding weight data of each input channel respectively to obtain the target characteristic value of the weight data of each output channel; according to the target characteristic value of the weight data of each output channel, the position of the weight data of each output channel in the weight matrix of any network layer to be quantized is transformed in the order from small to large to obtain the weight matrix to be quantized of any network layer to be quantized, and the transformation matrix of the weight matrix to be quantized of any network layer to be quantized relative to the weight matrix of any network layer to be quantized is used as the first transformation matrix.

[0020] According to another aspect of the embodiments of the present disclosure, a quantization system for a neural network model is provided, comprising a storage-computing integrated accelerator and a quantization device for the neural network model described in any of the above embodiments of the present disclosure.

[0021] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0022] processor;

[0023] a memory for storing instructions executable by the processor;

[0024] The processor is used to read the executable instructions from the memory and execute the instructions to implement the neural network quantization method described in any of the above embodiments of the present disclosure.

[0025] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the neural network quantization method described in any of the above embodiments of the present disclosure.

[0026] Based on the quantization method, device and system, electronic device and medium of the neural network model provided by the above embodiments of the present disclosure, by quantizing the weight matrix after row rearrangement and / or column rearrangement of any network layer to be quantized in the neural network model to be quantized, the distribution difference of weight data of each channel in the weight matrix can be reduced, thereby reducing the quantization error and helping to improve the accuracy of the neural network after quantization.

[0027] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0030] Figure 1 The present invention is a flowchart of an embodiment of a quantization method for a neural network model disclosed herein.

[0031] Figure 2 This is a schematic diagram of mapping the weight matrix to the storage and computing integrated accelerator in the present invention.

[0032] Figure 3 It is a schematic diagram of the multi-layer joint matrix transformation disclosed in the present invention.

[0033] Figure 4 The present invention is a flowchart of another embodiment of the quantization method of the neural network model disclosed in the present invention.

[0034] Figure 5 The figure is a schematic diagram of the structure of an embodiment of a quantization device for a neural network model disclosed in the present invention.

[0035] Figure 6 It is a structural schematic diagram of another embodiment of the quantization device of the neural network model disclosed in the present invention.

[0036] Figure 7 The figure is a schematic diagram of the structure of an embodiment of a quantization system of a neural network model disclosed in the present invention.

[0037] Figure 8The figure is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0038] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0039] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0040] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0041] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0042] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0043] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0044] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0046] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0047] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0048] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0049] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0050] Figure 1 FIG. 1 is a flow chart of an embodiment of a quantization method of a neural network model disclosed in the present invention. Figure 1 As shown, this embodiment includes the following steps:

[0051] Step 102, respectively, for any network layer to be quantized in the neural network model to be quantized, obtain a weight matrix of any network layer to be quantized.

[0052] In the disclosed embodiment, the neural network model to be quantized can be a neural network model that has not been pre-trained, or a neural network model that has been pre-selected for training, which is not limited in the disclosed embodiment. In the case where the neural network model to be quantized is a pre-trained neural network model, the disclosed embodiment can also pre-train the established neural network model with training data to obtain a pre-trained neural network model as the neural network model to be quantized, and the specific training process will not be repeated.

[0053] In the disclosed embodiment, the neural network model to be quantized may be any one of convolutional neural networks such as LeNet, AlexNet, VggNet and ResNet. The network structure of the neural network model to be quantized may include an input layer, a hidden layer and an output layer, wherein the hidden layer may include but is not limited to a convolutional layer, an excitation layer, a pooling layer and a fully connected layer. The disclosed embodiment does not limit the type and network structure of the neural network model to be quantized, nor does it limit the number of convolutional layers and the number of fully connected layers in the neural network model to be quantized.

[0054] The network layer to be quantized in the embodiment of the present disclosure may be any one or more network layers outside the input layer and the output layer in the neural network model to be quantized, such as any convolutional layer or any fully connected layer. The embodiment of the present disclosure does not limit the type and number of network layers to be quantized in the neural network model to be quantized.

[0055] Step 104 , performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized.

[0056] In the disclosed embodiment, a row of weight data in the weight matrix of any network layer to be quantized may correspond to the weight of each input of a certain neuron in the network layer to be quantized, or may correspond to the weight of each neuron in the quantized neural network layer for a certain input. Correspondingly, a column of weight data in the weight matrix of any network layer to be quantized may correspond to the weight of each neuron in the quantized neural network layer for a certain input, or may correspond to the weight of each neuron in the network layer to be quantized for a certain input.

[0057] The matrix transformation (also referred to as matrix rearrangement) in the embodiments of the present disclosure may be a row transformation (also referred to as row rearrangement) and / or a column transformation (also referred to as column rearrangement) of the weight matrix, wherein the row transformation may be a position transformation of the rows in the weight matrix, and the column transformation may be a position transformation of the columns in the weight matrix.

[0058] In an optional example, rows and / or columns in a weight matrix of any network layer to be quantized may be randomly selected for transformation.

[0059] In an optional example, the row transformation method of the weight matrix can be determined according to the distribution difference of each row of weight data in the weight matrix of any network layer to be quantized, and the column transformation method of the weight matrix can be determined according to the difference of each column of weight data in the weight matrix of any network layer to be quantized. By transforming the rows and / or columns of the weight matrix, the distribution difference of each row of weight data and each column of weight data in the weight matrix can be reduced, thereby improving the quantization accuracy of the network model.

[0060] Step 106, quantize the weight matrix to be quantized of any network layer to be quantized to obtain a quantized weight matrix of any network layer to be quantized.

[0061] Through the quantization algorithm, the weight data of the neural network can be converted from floating point numbers to fixed point numbers with lower bits. In the embodiment of the present disclosure, the quantized weight matrix is ​​obtained by quantizing the weight matrix to be quantized, and the weight data in the quantized weight matrix can be quantized into fixed point data with lower bits. When quantizing the weight matrix to be quantized, a symmetric uniform quantization form can be adopted, and its quantization function can be:

[0062]

[0063] Among them, x int is the fixed-point data after quantization, x is the floating-point data before quantization, Δ is the quantization scaling factor, k is the number of quantized bits, and the value range after k-bit quantization is [-2 k-1 ,2 k-1 -1].

[0064] Among them, the round() function is used to return a floating point number The clamp() function is used to limit the value to a given interval [min, max]. Values ​​less than the minimum value min can be mapped to min, and values ​​greater than the maximum value max can be mapped to max.

[0065] In the disclosed embodiment, the quantization granularity of the neural network model to be quantized can determine which data share a quantization scaling factor. If quantization is performed with the layer as the quantization granularity, the weight data of each layer shares a scaling factor; if quantization is performed with the output channel as the quantization granularity, the weight data of each output channel shares a scaling factor.

[0066] Step 108, obtaining a quantized neural network model based on the quantized weight matrix of any network layer to be quantized in the neural network model to be quantized.

[0067] In the disclosed embodiment, any of the multiple network layers to be quantized in the neural network model to be quantized can be quantized respectively, and after completing the quantization of the multiple network layers to be quantized, a quantized neural network model can be obtained.

[0068] The inventors of the disclosed embodiments can use the ResNet-20 network as the neural network model to be quantized, use the CIFAR-10 data set as the test set, randomly rearrange the rows and columns of each network layer to be quantized, use 4-bit quantization for each network layer to be quantized (the quantization tool can be EasyQuant), and then statistically analyze the accuracy of the neural network model after quantization. The accuracy of the quantized neural network varies greatly depending on the rearrangement. The difference between the accuracy of the best rearrangement and the accuracy of the worst rearrangement is as high as 2%. Therefore, weight rearrangement can greatly improve the accuracy of the neural network after the quantization model.

[0069] Based on the quantization method of the neural network model provided by the above-mentioned embodiment of the present disclosure, by quantizing the weight matrix after row rearrangement and / or column rearrangement of any network layer to be quantized in the neural network model to be quantized, the distribution difference of the weight data of each row and each column in the weight matrix can be reduced, thereby reducing the quantization error and helping to improve the accuracy of the neural network after quantization.

[0070] Optionally, in some possible implementations of the present disclosure, in the above step 106, when quantizing the weight matrix to be quantized of any network layer to be quantized to obtain the quantized weight matrix of any network layer to be quantized, the weight matrix to be quantized of any network layer to be quantized can be divided according to the size of the storage and computing unit array in the storage and computing integrated accelerator to obtain multiple sub-weight matrices to be quantized, and then any sub-weight matrix to be quantized in the multiple sub-weight matrices to be quantized is quantized respectively to obtain multiple quantized sub-weight matrices, and the multiple quantized sub-weight matrices are respectively mapped to multiple storage and computing unit arrays in the storage and computing integrated accelerator, and the multiple quantized sub-weight matrices are stored by the multiple storage and computing unit arrays.

[0071] Figure 2 This is a schematic diagram of mapping the weight matrix to the storage and computing integrated accelerator in the present disclosure. Figure 2 As shown, W is an n×m weight matrix, and each rectangle on the right side of the arrow is an R×C storage and computing unit array. In the storage and computing integrated accelerator, the weight matrix W can be mapped to #R×#C R×C storage and computing unit arrays, where

[0072] Based on this, the embodiments of the present disclosure can divide the weight matrix to be quantized of any network layer to be quantized according to the size of the storage and computing unit array in the storage and computing integrated accelerator to obtain multiple sub-weight matrices to be quantized, and then quantize any sub-weight matrix to be quantized in the multiple sub-weight matrices to be quantized respectively, each sub-weight matrix to be quantized shares a quantization scaling coefficient, and obtain multiple quantized sub-weight matrices, and then the multiple quantized sub-weight matrices can be mapped to the multiple storage and computing unit arrays in the storage and computing integrated accelerator respectively, and the multiple storage and computing unit arrays store the multiple quantized sub-weight matrices.

[0073] The disclosed embodiment divides the weight matrix to be quantized of the network layer to be quantized into multiple sub-weight matrices to be quantized according to the size of the storage and computing unit array, and then quantizes each weight matrix to be quantized respectively, which helps to reduce the distribution difference of weight data stored in each storage and computing unit array, helps to reduce quantization errors, and improves the quantization accuracy of the network model.

[0074] Optionally, in some possible implementations of the present disclosure, after the above step 104, the weight matrix to be quantized may be optimized by the following steps:

[0075] Step a, initializing the number of matrix transformations to a preset value.

[0076] In the embodiment of the present disclosure, the initial value of the number of matrix transformations is a preset value, which can be a value such as 0 or 1. The maximum value of the number of matrix transformations is a preset threshold, which can be set according to actual needs and is not limited in the embodiment of the present disclosure.

[0077] Step b, randomly select any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized and exchange them to obtain a new weight matrix to be quantized of any network layer to be quantized, and then quantize the new weight matrix to be quantized of any network layer to be quantized to obtain a new quantized weight matrix of any network layer to be quantized, and based on the new quantized weight matrix of any network layer to be quantized, obtain a new quantized neural network model.

[0078] In an embodiment of the present disclosure, when quantizing a new weight matrix to be quantized of any network layer to be quantized, the new weight matrix to be quantized of any network layer to be quantized can be mapped to multiple storage and computing unit arrays in a storage and computing integrated accelerator, and the storage and computing integrated accelerator quantizes the weight data stored in any storage and computing unit array in the multiple storage and computing unit arrays to obtain a new quantized weight matrix of any network layer to be quantized.

[0079] Step c, determining the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized, and determining whether the cosine distance is less than a preset cosine distance, and determining whether the number of matrix transformations is less than a preset threshold.

[0080] In the disclosed embodiment, the output of the new quantized neural network model is to provide multiple calibration samples in the calibration sample set as input, respectively, to the new quantized neural network model, and the output of the new quantized neural network model is to process the multiple calibration samples through the new quantized neural network model, and the output of the neural network model to be quantized is to provide multiple calibration samples as input, respectively, to the neural network model to be quantized, and the output of the multiple calibration samples is obtained by processing the multiple calibration samples through the neural network model to be quantized.

[0081] In the embodiment of the present disclosure, the formula for determining the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized may be:

[0082]

[0083] Among them, Y 2 The output of the new quantized neural network model, Y 1 is the output of the neural network model to be quantized.

[0084] After using formula (2) to determine the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized, it can be determined whether the cosine distance is less than the preset cosine distance, and whether the number of matrix transformations is less than the preset threshold. The preset cosine distance can be determined according to actual needs and is not limited in the embodiments of the present disclosure.

[0085] Step d: If the cosine distance is less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, the target probability can be determined based on the cosine distance and the preset cosine distance, and it is determined whether the target probability is greater than the preset probability.

[0086] In the embodiment of the present disclosure, the formula for determining the target probability based on the cosine distance and the preset cosine distance may be:

[0087]

[0088] Where exp() is an exponential function with the natural constant e as the base, P is the target probability, and cossim i is the cosine distance, cossim i-1 is the preset cosine distance, and T is a hyperparameter of the simulated annealing algorithm. It should be noted that the value of T can be set according to actual needs, and the embodiment of the present disclosure does not limit it.

[0089] In step e, if the target probability is greater than the preset probability, the cosine distance can be used as the preset cosine distance, the number of matrix transformations is increased by 1, and an operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed.

[0090] Step f: If the target probability is not greater than the preset probability, the new weight matrix to be quantized of any network layer to be quantized can be used as the weight matrix to be quantized of any network layer to be quantized.

[0091] Alternatively, in some other possible implementations of the present disclosure, the step of optimizing the weight matrix to be quantized may further include:

[0092] Step g: If the cosine distance is less than the preset cosine distance and the number of matrix transformations is not less than the preset threshold, the new weight matrix to be quantized of any network layer to be quantized can be used as the weight matrix to be quantized of any network layer to be quantized.

[0093] Step h, if the cosine distance is not less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, the cosine distance can be used as the preset cosine distance, the number of matrix exchanges is increased by 1, and an operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed.

[0094] The inventors of the embodiments of the present disclosure can use the ResNet-20 network as the neural network model to be quantized, use the CIFAR-10 data set as the test set, optimize the weight matrix to be quantized of each network layer to be quantized according to steps a to h, use 4-bit quantization on each network layer to be quantized (the quantization tool can be EasyQuant), and then statistically analyze the accuracy of the neural network model after quantization. It is found that optimizing the weight matrix to be quantized of each network layer to be quantized according to steps a to h can effectively improve the cosine similarity between the quantized neural network model and the neural network model to be quantized, and can ultimately achieve a higher quantization accuracy of the neural network model.

[0095] The disclosed embodiment further adjusts the distribution difference of row weight data or column weight data in the weight matrix to be quantized by exchanging any two rows or two columns randomly selected from the weight matrix to be quantized of any network layer to be quantized, and determines whether the new weight matrix to be quantized can be used as the weight matrix to be quantized of any network layer to be quantized through the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized and the number of matrix transformations, which can further reduce the distribution difference of row weight data and column weight data in the weight matrix and improve the quantization accuracy of the network model.

[0096] Optionally, in some possible implementations of the present disclosure, in the above step 104, when performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized, a joint network layer of any network layer to be quantized can be determined, and the joint network layer is any other network layer to be quantized except any network layer to be quantized in the neural network model to be quantized, and the joint network layer takes the output of any network layer to be quantized as input, and then a first transformation matrix of the weight matrix of any network layer to be quantized can be determined, and the first transformation matrix is ​​used to perform matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized, and then the transposed matrix of the first transformation matrix can be used as the second transformation matrix of the weight matrix of the joint network layer, and the second transformation matrix is ​​used to perform matrix transformation on the weight matrix of the joint network layer to obtain the weight matrix to be quantized of the joint network layer.

[0097] In the disclosed embodiments, a column of weight data in the weight matrix of any network layer to be quantized may correspond to the weight of each input of a certain neuron in the network layer to be quantized, and a row of weight data in the weight matrix of any network layer to be quantized may correspond to the weight of each neuron in the quantized neural network layer for a certain input. A column of weight data in the weight matrix of any network layer to be quantized may be used as the weight data of an output channel, and a row of weight data in the weight matrix of any network layer to be quantized may be used as the weight data of an input channel. Exchanging the columns of the weight matrix may allow the positions of the output channels to be exchanged, and exchanging the rows of the weight matrix may allow the positions of the input channels to be exchanged.

[0098] In the disclosed embodiment, if the weight matrix of any network layer to be quantized is rearranged by row transformation, or the weight matrix of any network layer to be quantized is rearranged by column exchange, when calculating the network layer to be quantized, it is necessary to obtain the input data of the corresponding position according to the index of the row rearrangement; the output of the network layer to be quantized also needs to store the calculation result in the corresponding position according to the index of the column rearrangement. Such an indexing method requires calculating the location of the data in the memory, and random access to the storage unit will also occur, resulting in a large overhead.

[0099] As an example, for any network layer C1 to be quantized in the neural network model to be quantized, the input data of C1 may include (d1, d2, d3), and C1 may include 3 neurons (s1, s2, s3). Then the weight matrix of C1 is shown in Table 1 below:

[0100]

[0101] Table 1 Weight matrix of the network layer C1 to be quantized

[0102] For the joint network layer C2 of any network layer C1 to be quantized, C2 takes the output of each neuron of C1 as input. If C2 includes 3 neurons (t1, t2, t3), then the weight matrix of C2 is as shown in the following Table 2:

[0103]

[0104] Table 2 Weight matrix of joint network layer C2

[0105] Comparing the weight matrices of C1 and C2, it is found that if the first row and the second row in the weight matrix of C1 are swapped, the first column and the second column in the weight matrix of C2 need to be swapped to achieve the position correspondence between the input channel of C2 and the output channel of C1.

[0106] Based on this, in the embodiment of the present disclosure, when quantizing any network layer to be quantized, any network layer to be quantized can be subjected to a joint matrix transformation (also referred to as a joint rearrangement) with a joint network layer that uses the output of any network layer to be quantized as input. Since the joint network layer uses the output of any network layer to be quantized as input, the output channel of any network layer to be quantized corresponds to the input channel of the joint network layer, and therefore the output channel position of any network layer to be quantized and the input channel position of the joint network layer can be adjusted simultaneously.

[0107] Figure 3 Schematic diagram of multi-layer joint matrix transformation disclosed in the present invention. Figure 3 As shown, W l is the weight matrix of the lth layer, W l+1 is the weight matrix of the l+1th layer, f is the activation function, and the activation function f can be a commonly used piecewise linear function such as ReLU, PReLU, ReLU6, etc., which is not limited in the embodiments of the present disclosure.

[0108] The batch-normalization calculation in the network is integrated into the weights. The calculations of the lth layer and the l+1th layer are:

[0109] X l+1 =f(W l X l ) (4)

[0110] X l+2 =f(W l+1 X l+1 ) (5)

[0111] The calculation formula for merging two layers is:

[0112] X l+2 =f(W l+1 f(W l Xl )) (6)

[0113] In order to eliminate the overhead of data access index, the lth layer and the l+1th layer are jointly rearranged. The columns of the lth layer are exchanged using the matrix S to obtain W l S, using its transposed matrix S T Swap the rows of layer l+1 to get S T W l+1 .like Figure 3 As shown:

[0114] Figure 3 Middle W l The i-th column and the j-th column of W are swapped, and l+1 Swap the i-th row with the j-th row.

[0115] Using the rearranged weights to directly calculate, we have:

[0116]

[0117] When the activation function f is a commonly used piecewise linear function such as ReLU, PReLU, ReLU6, etc., the following equation is obtained:

[0118]

[0119] It can be seen that the calculation results after joint rearrangement The result before joint rearrangement is X l+2 Likewise, the overhead of data fetch indexes can be avoided.

[0120] In an optional example, when the lth layer and the l+1th layer are jointly rearranged, the matrix S can also be used to exchange the rows of the lth layer to obtain SW l , using its transposed matrix S T Swap the columns of layer l+1 to get W l+1 S T , the specific joint rearrangement process will not be repeated.

[0121] The inventors of the disclosed embodiments can use the ResNet-20 network as the neural network model to be quantized, use the CIFAR-10 dataset as the test dataset, use joint random rearrangement on each layer of the neural network model to be quantized, and after 4-bit quantization (the quantization tool can be EasyQuant), use the statistical model to predict the accuracy, and find that the accuracy of the jointly rearranged quantized neural network is close to the accuracy of the quantized neural network after each layer is rearranged separately. Compared with the separate rearrangement of each layer, multiple joint rearrangements can eliminate the overhead of data access index and can greatly improve the quantization accuracy of the network model.

[0122] Optionally, in some possible implementations of the present disclosure, in the above step 104, when determining the first transformation matrix of the weight matrix of any network layer to be quantized, the correspondence between the weight data of each output channel in the weight matrix of any network layer to be quantized and the weight data of each input channel in the weight matrix of the joint network layer can be determined based on the correspondence between the input channel of the joint network layer and the output channel of any network layer to be quantized, and then the distribution characteristic value of the weight data of each output channel and the distribution characteristic value of the weight data of each input channel can be determined, and the distribution characteristic value of the weight data of each output channel can be multiplied by the distribution characteristic value of the weight data of each corresponding input channel, respectively, to obtain the target characteristic value of the weight data of each output channel; and then the weight data of each output channel in the weight matrix of any network layer to be quantized can be transformed in position according to the target characteristic value of the weight data of each output channel from small to large, to obtain the weight matrix to be quantized of any network layer to be quantized, and the transformation matrix of the weight matrix to be quantized of any network layer to be quantized relative to the weight matrix of any network layer to be quantized is used as the first transformation matrix.

[0123] In the disclosed embodiment, the distribution characteristic value of the weight data may be the range between the 99.9% and 0.1% quantiles of the weight data. The weight data may be sorted in order from small to large, and the weight value corresponding to 99.9% of the weight data may be determined to be the 99.9% quantile, and the weight value corresponding to 0.1% of the weight data may be determined to be the 0.1% quantile.

[0124] As an example, suppose is the weight range of the i-th output channel of the l-th layer network, is the weight range of the i-th input channel of the l+1-th layer network. Sort the output channels of the lth layer network and the input channels of the l+1th layer network, The output channel or input channel with a smaller value is rearranged to the front. Output channels or input channels with larger values ​​are rearranged to the back. This sorting takes into account the weight data distribution of the weight matrix of the two-layer network, so that when the weight matrix of the lth layer and the weight matrix of the l+1th layer are mapped to the storage-computation integrated accelerator, the distribution of the weight data stored in the same storage-computation unit array is closer, which helps to reduce quantization errors and improve the quantization accuracy of the network model.

[0125] Figure 4 FIG. 1 is a flow chart of another embodiment of the quantization method of the neural network model disclosed in the present invention. Figure 4 As shown, in Figure 1 Based on the illustrated embodiment, after the above step 108, the above method may further include:

[0126] Step 110, obtaining multiple calibration samples from the calibration sample set, providing the multiple calibration samples as input to the quantized neural network model respectively, and processing the input multiple calibration samples through the quantized neural network model to obtain the output of the quantized neural network model.

[0127] Step 112, according to the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized, adjust the weight matrix to be quantized of any network layer to be quantized. The output of the neural network model to be quantized is the output obtained by processing the input multiple calibration samples by providing them to the neural network model to be quantized.

[0128] In the disclosed embodiment, the calibration samples in the calibration sample set may be samples without labels, so the accuracy of the quantized neural network model cannot be measured based on the labels of the calibration samples. In order to measure the accuracy of the quantized neural network model, the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized may be used as a target for measuring the accuracy of the quantized neural network model.

[0129] The purpose of quantizing a neural network model is to make the output of the quantized neural network model as close as possible to the output of the neural network model to be quantized. Therefore, the matrix rearrangement goal of the weight matrix is ​​to minimize the difference between the output of the quantized neural network model and the output of the neural network model to be quantized. The matrix rearrangement goal can be defined as:

[0130]

[0131] where Y quant is the output value of the quantized neural network model; Y FP32 It is the output value of the neural network model to be quantized. FP32 is a data type that uses 4 bytes (32 bits) for encoding and storage; S row ,S col is the transformation matrix for the weight matrix; cossim is the function for calculating the average cosine distance of different calibration inputs, defined as follows:

[0132]

[0133] The inventors of the embodiments of the present disclosure can use the ResNet-20 network as the neural network model to be quantized, use the CIFAR-10 data set as the calibration sample set, use 4-bit quantization on each layer of the neural network model to be quantized (the quantization tool can be EasyQuant), and then count the accuracy of the neural network model after quantization. It is found that by adjusting the weight matrix to be quantized of any network layer to be quantized according to the cosine distance between the output of the neural network model after quantization and the output of the neural network model to be quantized, the output of the neural network model after quantization is closer to the output of the network model to be quantized, which can improve the prediction accuracy of the neural network model after quantization while ensuring that the number of bits of the quantized weight data is low.

[0134] Figure 5 FIG. 1 is a schematic diagram of a structure of an embodiment of a quantization device for a neural network model of the present disclosure. The quantization device for a neural network module of this embodiment can be used to implement the quantization method embodiments of each neural network model of the present disclosure. Figure 5 As shown, the device of this embodiment may include: an acquisition module 502, a matrix transformation module 504, a quantization module 506 and a first obtaining module 508.

[0135] The acquisition module 502 is used to acquire the weight matrix of any network layer to be quantized in the neural network model to be quantized.

[0136] The matrix transformation module 504 is used to perform matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized.

[0137] The quantization module 506 is used to quantize the weight matrix to be quantized of any network layer to be quantized to obtain a quantized weight matrix of any network layer to be quantized.

[0138] The first obtaining module 508 is used to obtain a quantized neural network model based on a quantized weight matrix of any network layer to be quantized in the neural network model to be quantized.

[0139] Optionally, in some possible implementations of the present disclosure, the above-mentioned quantization module 506 may include: a division unit, a quantization unit, and a transmission unit. Among them, the division unit is used to divide the weight matrix to be quantized of any network layer to be quantized according to the size of the storage and computing unit array in the storage and computing integrated accelerator to obtain multiple sub-weight matrices to be quantized; the quantization unit is used to quantize any sub-weight matrix to be quantized in the multiple sub-weight matrices to be quantized, respectively, to obtain multiple quantized sub-weight matrices; the mapping unit is used to map the multiple quantized sub-weight matrices to the multiple storage and computing unit arrays in the storage and computing integrated accelerator, and the multiple storage and computing unit arrays store the multiple quantized sub-weight matrices.

[0140] Optionally, in some possible implementations of the present disclosure, the above-mentioned matrix transformation module 504 can also be used to: initialize the number of matrix transformations to a preset value; randomly select any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange, to obtain a new weight matrix to be quantized of any network layer to be quantized; quantize the new weight matrix to be quantized of any network layer to be quantized, to obtain a new quantized weight matrix of any network layer to be quantized, and based on the new quantized weight matrix of any network layer to be quantized, obtain a new quantized neural network model; determine the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized , and determine whether the cosine distance is less than the preset cosine distance, and determine whether the number of matrix transformations is less than the preset threshold; if the cosine distance is less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, determine the target probability based on the cosine distance and the preset cosine distance, and determine whether the target probability is greater than the preset probability; if the target probability is greater than the preset probability, use the cosine distance as the preset cosine distance, increase the number of matrix transformations by 1, and perform an operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange; if the target probability is not greater than the preset probability, use the new weight matrix to be quantized of any network layer to be quantized as the weight matrix to be quantized of any network layer to be quantized.

[0141] Optionally, in some possible implementations of the present disclosure, the above-mentioned matrix transformation module 504 can also be used for: if the cosine distance is less than the preset cosine distance and the number of matrix transformations is not less than a preset threshold, the new weight matrix to be quantized of any network layer to be quantized is used as the weight matrix to be quantized of any network layer to be quantized; if the cosine distance is not less than the preset cosine distance and the number of matrix transformations is less than a preset threshold, the cosine distance is used as the preset cosine distance, the number of matrix exchanges is increased by 1, and an operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed.

[0142] Optionally, in some possible implementations of the present disclosure, the matrix transformation module 504 may include: a determination unit, a first transformation unit, and a second transformation unit.

[0143] A determination unit is used to determine a joint network layer of any network layer to be quantized, where the joint network layer is any other network layer to be quantized in the neural network model to be quantized except any network layer to be quantized, and the joint network layer uses the output of any network layer to be quantized as input; a first transformation unit is used to determine a first transformation matrix of a weight matrix of any network layer to be quantized, and use the first transformation matrix to perform a matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; a second transformation unit is used to use the transposed matrix of the first transformation matrix as a second transformation matrix of the weight matrix of the joint network layer, and use the second transformation matrix to perform a matrix transformation on the weight matrix of the joint network layer to obtain the weight matrix to be quantized of the joint network layer.

[0144] Optionally, in some possible implementations of the present disclosure, when the matrix transformation module 504 determines the first transformation matrix of the weight matrix of any network layer to be quantized, it can determine the correspondence between the weight data of each output channel in the weight matrix of any network layer to be quantized and the weight data of each input channel in the weight matrix of the joint network layer based on the correspondence between the input channel of the joint network layer and the output channel of any network layer to be quantized; determine the distribution characteristic value of the weight data of each output channel and the distribution characteristic value of the weight data of each input channel, and multiply the distribution characteristic value of the weight data of each output channel by the distribution characteristic value of the corresponding weight data of each input channel, respectively, to obtain the target characteristic value of the weight data of each output channel; according to the target characteristic value of the weight data of each output channel, the position of the weight data of each output channel in the weight matrix of any network layer to be quantized is transformed in the order from small to large, to obtain the weight matrix to be quantized of any network layer to be quantized, and the transformation matrix of the weight matrix to be quantized of any network layer to be quantized relative to the weight matrix of any network layer to be quantized is used as the first transformation matrix.

[0145] Figure 6 FIG. 1 is a schematic diagram of the structure of another embodiment of the quantization device of the neural network model disclosed in the present invention. The quantization device of the neural network model of the another embodiment, Figure 6 Based on the quantization device of the neural network model shown, a second obtaining module 510 and an adjusting module 512 may also be included.

[0146] The second obtaining module 510 is used to obtain multiple calibration samples from the calibration sample set, provide the multiple calibration samples as input to the quantized neural network model respectively, and process the input multiple calibration samples through the quantized neural network model to obtain the output of the quantized neural network model.

[0147] The adjustment module 512 is used to adjust the weight matrix to be quantized of any network layer to be quantized according to the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized. The output of the neural network model to be quantized is to provide multiple calibration samples as input to the neural network model to be quantized, and the output is obtained by processing the input multiple calibration samples by the neural network model to be quantized.

[0148] Figure 7 The schematic diagram of the structure of an embodiment of the quantization system of the neural network model of the present disclosure is shown in FIG. The quantization system of the neural network model may include a storage-computation integrated accelerator and a quantization device of the neural network model described in any of the above embodiments of the present disclosure.

[0149] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0150] Memory for storing computer programs;

[0151] A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the quantization method of the neural network model described in any of the above embodiments of the present disclosure is implemented.

[0152] Figure 8 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 8 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0153] like Figure 8 As shown, the electronic device includes one or more processors and memory.

[0154] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0155] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the quantization method of the neural network model of each embodiment of the present disclosure described above and / or other desired functions.

[0156] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0157] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0158] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0159] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0160] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the quantization method of the neural network model according to various embodiments of the present disclosure described in the above part of this specification.

[0161] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0162] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the quantization method of the neural network model according to various embodiments of the present disclosure described in the above part of this specification.

[0163] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0165] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0166] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0168] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0169] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0170] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0171] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A quantization method for a neural network model, characterized in that: The method is applied to a storage-computing integrated accelerator, and the method includes: For each network layer to be quantized in the neural network model to be quantized, a weight matrix of the network layer to be quantized is obtained; Performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; Quantizing the weight matrix to be quantized of any network layer to be quantized to obtain a quantized weight matrix of any network layer to be quantized; Based on the quantized weight matrix of any network layer to be quantized in the neural network model to be quantized, a quantized neural network model is obtained; The step of quantizing the weight matrix to be quantized of any network layer to be quantized to obtain the quantized weight matrix of any network layer to be quantized includes: According to the size of the storage and computing unit array in the storage and computing integrated accelerator, the weight matrix to be quantized of any network layer to be quantized is divided to obtain a plurality of sub-weight matrices to be quantized; Based on the quantization scaling coefficient shared by the multiple sub-weight matrices to be quantized, quantize any sub-weight matrix to be quantized in the multiple sub-weight matrices to be quantized respectively to obtain multiple quantized sub-weight matrices; The multiple quantized sub-weight matrices are respectively mapped to multiple storage and computing unit arrays in the storage and computing integrated accelerator, and the multiple storage and computing unit arrays store the multiple quantized sub-weight matrices.

2. The method according to claim 1, characterized in that After obtaining the quantized neural network model based on the quantized weight matrix of any network layer to be quantized in the neural network model to be quantized, the method further includes: Acquire multiple calibration samples from the calibration sample set, provide the multiple calibration samples as input to the quantized neural network model respectively, and process the input multiple calibration samples through the quantized neural network model to obtain the output of the quantized neural network model; According to the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized, the weight matrix to be quantized of any network layer to be quantized is adjusted; wherein the output of the neural network model to be quantized is the output obtained by providing the multiple calibration samples as input to the neural network model to be quantized, and processing the input multiple calibration samples by the neural network model to be quantized.

3. The method according to claim 1 or 2, characterized in that: After performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized, the method further includes: Initialize the number of matrix transformations to a preset value; Randomly select any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized and exchange them to obtain a new weight matrix to be quantized of any network layer to be quantized; quantize the new weight matrix to be quantized of any network layer to be quantized to obtain a new quantized weight matrix of any network layer to be quantized, and obtain a new quantized neural network model based on the new quantized weight matrix of any network layer to be quantized; Determine the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized, determine whether the cosine distance is less than a preset cosine distance, and determine whether the number of matrix transformations is less than a preset threshold; If the cosine distance is less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, determining a target probability based on the cosine distance and the preset cosine distance, and determining whether the target probability is greater than a preset probability; If the target probability is greater than the preset probability, the cosine distance is used as the preset cosine distance, the number of matrix transformations is increased by 1, and the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed; If the target probability is not greater than the preset probability, the new weight matrix to be quantized of any network layer to be quantized is used as the weight matrix to be quantized of any network layer to be quantized.

4. The method according to claim 3, characterized in that: Also includes: If the cosine distance is less than the preset cosine distance and the number of matrix transformations is not less than the preset threshold, using the new weight matrix to be quantized of any network layer to be quantized as the weight matrix to be quantized of any network layer to be quantized; If the cosine distance is not less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, the cosine distance is used as the preset cosine distance, the number of matrix exchanges is increased by 1, and the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed.

5. The method according to claim 3, characterized in that: The performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized includes: Determine a joint network layer of any of the network layers to be quantized, where the joint network layer is any other network layer to be quantized in the neural network model to be quantized except the any of the network layers to be quantized, and the joint network layer uses the output of the any of the network layers to be quantized as input; Determine a first transformation matrix of the weight matrix of any network layer to be quantized, and use the first transformation matrix to perform matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; The transposed matrix of the first transformation matrix is ​​used as the second transformation matrix of the weight matrix of the joint network layer, and the second transformation matrix is ​​used to perform matrix transformation on the weight matrix of the joint network layer to obtain the weight matrix to be quantized of the joint network layer.

6. The method according to claim 5, characterized in that The determining of the first transformation matrix of the weight matrix of any network layer to be quantized comprises: Based on the correspondence between the input channel of the joint network layer and the output channel of any of the network layers to be quantized, determining the correspondence between the weight data of each output channel in the weight matrix of any of the network layers to be quantized and the weight data of each input channel in the weight matrix of the joint network layer; Determine the distribution characteristic value of the weight data of each output channel and the distribution characteristic value of the weight data of each input channel, and multiply the distribution characteristic value of the weight data of each output channel by the distribution characteristic value of the corresponding weight data of each input channel to obtain the target characteristic value of the weight data of each output channel; According to the target eigenvalues ​​of the weight data of each output channel in the weight matrix of any network layer to be quantized, the position of the weight data of each output channel is transformed to obtain the weight matrix to be quantized of any network layer to be quantized, and the transformation matrix of the weight matrix to be quantized of any network layer to be quantized relative to the weight matrix of any network layer to be quantized is used as the first transformation matrix.

7. A quantization device for a neural network model, characterized in that: The device is applied to a storage-computing integrated accelerator, and the device includes: An acquisition module, used for acquiring a weight matrix of any network layer to be quantized in the neural network model to be quantized; A matrix transformation module, used for performing matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; A quantization module, used for quantizing the weight matrix to be quantized of any network layer to be quantized to obtain a quantized weight matrix of any network layer to be quantized; A first obtaining module is used to obtain a quantized neural network model based on a quantized weight matrix of any network layer to be quantized in the neural network model to be quantized; The quantization module comprises: A division unit, configured to divide the weight matrix to be quantized of any network layer to be quantized according to the size of the storage and computing unit array in the storage and computing integrated accelerator, to obtain a plurality of sub-weight matrices to be quantized; A quantization unit, configured to quantize any sub-weight matrix to be quantized among the multiple sub-weight matrices to be quantized based on a quantization scaling coefficient shared by the multiple sub-weight matrices to be quantized, to obtain multiple quantized sub-weight matrices; A mapping unit is used to map the multiple quantized sub-weight matrices to multiple storage and computing unit arrays in the storage and computing integrated accelerator respectively, and the multiple storage and computing unit arrays store the multiple quantized sub-weight matrices.

8. The device according to claim 7, characterized in that Also includes: A second obtaining module is used to obtain a plurality of calibration samples from the calibration sample set, provide the plurality of calibration samples as input to the quantized neural network model respectively, and process the input plurality of calibration samples through the quantized neural network model to obtain the output of the quantized neural network model; An adjustment module is used to adjust the weight matrix to be quantized of any network layer to be quantized according to the cosine distance between the output of the quantized neural network model and the output of the neural network model to be quantized; wherein the output of the neural network model to be quantized is to provide the multiple calibration samples as input to the neural network model to be quantized, and the output is obtained by processing the input multiple calibration samples through the neural network model to be quantized.

9. The device according to claim 7 or 8, characterized in that The matrix transformation module is also used for: Initialize the number of matrix transformations to a preset value; Randomly select any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized and exchange them to obtain a new weight matrix to be quantized of any network layer to be quantized; Quantizing a new weight matrix to be quantized of any network layer to be quantized to obtain a new quantized weight matrix of any network layer to be quantized, and obtaining a new quantized neural network model based on the new quantized weight matrix of any network layer to be quantized; Determine the cosine distance between the output of the new quantized neural network model and the output of the neural network model to be quantized, determine whether the cosine distance is less than a preset cosine distance, and determine whether the number of matrix transformations is less than a preset threshold; If the cosine distance is less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, determining a target probability based on the cosine distance and the preset cosine distance, and determining whether the target probability is greater than a preset probability; If the target probability is greater than the preset probability, the cosine distance is used as the preset cosine distance, the number of matrix transformations is increased by 1, and the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed; If the target probability is not greater than the preset probability, the new weight matrix to be quantized of any network layer to be quantized is used as the weight matrix to be quantized of any network layer to be quantized.

10. The device according to claim 9, characterized in that The matrix transformation module is also used for: If the cosine distance is less than the preset cosine distance and the number of matrix transformations is not less than the preset threshold, using the new weight matrix to be quantized of any network layer to be quantized as the weight matrix to be quantized of any network layer to be quantized; If the cosine distance is not less than the preset cosine distance and the number of matrix transformations is less than the preset threshold, the cosine distance is used as the preset cosine distance, the number of matrix exchanges is increased by 1, and the operation of randomly selecting any two rows or two columns in the weight matrix to be quantized of any network layer to be quantized for exchange is performed.

11. The device according to claim 9, characterized in that The matrix transformation module comprises: A determination unit, used to determine a joint network layer of any network layer to be quantized, wherein the joint network layer is any other network layer to be quantized in the neural network model to be quantized except the any network layer to be quantized, and the joint network layer uses the output of the any network layer to be quantized as input; A first transformation unit is used to determine a first transformation matrix of the weight matrix of any network layer to be quantized, and use the first transformation matrix to perform a matrix transformation on the weight matrix of any network layer to be quantized to obtain the weight matrix to be quantized of any network layer to be quantized; The second transformation unit is used to use the transposed matrix of the first transformation matrix as the second transformation matrix of the weight matrix of the joint network layer, and use the second transformation matrix to perform matrix transformation on the weight matrix of the joint network layer to obtain the weight matrix to be quantized of the joint network layer.

12. The device according to claim 11, characterized in that The determining of the first transformation matrix of the weight matrix of any network layer to be quantized comprises: Based on the correspondence between the input channel of the joint network layer and the output channel of any of the network layers to be quantized, determining the correspondence between the weight data of each output channel in the weight matrix of any of the network layers to be quantized and the weight data of each input channel in the weight matrix of the joint network layer; Determine the distribution characteristic value of the weight data of each output channel and the distribution characteristic value of the weight data of each input channel, and multiply the distribution characteristic value of the weight data of each output channel by the distribution characteristic value of the corresponding weight data of each input channel to obtain the target characteristic value of the weight data of each output channel; According to the target eigenvalues ​​of the weight data of each output channel in the weight matrix of any network layer to be quantized, the position of the weight data of each output channel is transformed to obtain the weight matrix to be quantized of any network layer to be quantized, and the transformation matrix of the weight matrix to be quantized of any network layer to be quantized relative to the weight matrix of any network layer to be quantized is used as the first transformation matrix.

13. A quantization system for a neural network model, characterized in that: A quantization device comprising a storage-computation integrated accelerator and a neural network model as described in any one of claims 7-12.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of claims 1 to 6.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images