Model data quantification method and device, electronic equipment and readable storage medium

Abstract

The invention provides a quantification method and device of model data, electronic equipment and a readable storage medium, and the quantification method of the model data comprises the steps: obtaining a first data matrix corresponding to the model data; generating a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and scaling the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used for constraining distribution characteristics of matrix element values in the second data matrix; and obtaining a quantification result of the model data according to the second data matrix. According to the mode, the data volume of the model data is reduced, the occupation of transmission bandwidth and the consumption of computing resources are reduced, and the processing efficiency of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for quantizing model data, an electronic device, and a readable storage medium. Background Art

[0002] Model data includes various types of data used to build and train models, as well as data required in the model reasoning process. Model data can be an abstract result of the data features in the model, used to describe the static characteristics, dynamic behaviors, and constraints of the system. It can be seen that model data plays a vital role in the model training and reasoning process.

[0003] In application scenarios with large amounts of data, the amount of model data is usually also large. Therefore, when model data is transmitted within the model, a large amount of transmission bandwidth and computing resources will be consumed, which will greatly affect the processing efficiency of the model. Therefore, how to reduce the amount of model data has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure provides a method and device for quantizing model data, an electronic device, and a readable storage medium.

[0005] In a first aspect, the present disclosure provides a method for quantifying model data, the method comprising:

[0006] Acquire a first data matrix corresponding to the model data;

[0007] According to preset constraints, a scaling matrix corresponding to the first data matrix is ​​generated, and the first data matrix is ​​scaled according to the scaling matrix to obtain a second data matrix; wherein the preset constraints are used to constrain the distribution characteristics of matrix element values ​​in the second data matrix;

[0008] A quantization result of the model data is obtained according to the second data matrix.

[0009] In a second aspect, the present disclosure provides a data processing method based on a neural network, comprising:

[0010] Obtaining model data corresponding to the first network layer of the neural network;

[0011] According to the above quantization method, a quantization result of the model data is obtained;

[0012] The quantization results and scaling matrix of the model data are transmitted to the second network layer of the neural network, so that the second network layer performs data processing according to the quantization results and scaling matrix of the model data.

[0013] In a third aspect, the present disclosure provides a device for quantizing model data, the device comprising:

[0014] A first acquisition module, adapted to acquire a first data matrix corresponding to the model data;

[0015] A generating module, adapted to generate a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and scale the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values ​​in the second data matrix;

[0016] The first quantization module is adapted to obtain a quantization result of the model data according to the second data matrix.

[0017] In a fourth aspect, the present disclosure provides a data processing device based on a neural network, comprising:

[0018] A second acquisition module, adapted to acquire model data corresponding to the first network layer of the neural network;

[0019] A second quantization module, adapted to obtain a quantization result of the model data according to the above quantization method;

[0020] The transmission module is suitable for transmitting the quantization results and scaling matrix of the model data to the second network layer of the neural network, so that the second network layer performs data processing according to the quantization results and scaling matrix of the model data.

[0021] In a fifth aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned method.

[0022] In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above method when executed by a processor / processing core.

[0023] In the embodiments provided in the present disclosure, the model data can be represented by a first data matrix. In order to reduce the data level of the first data matrix, in the present disclosure, a scaling matrix corresponding to the first data matrix is ​​generated according to preset constraints, and a second data matrix after scaling the first data matrix is ​​obtained according to the scaling matrix. Since the scaling matrix is ​​generated according to the preset constraints, the distribution characteristics of each matrix element value in the second data matrix obtained after scaling can be better controlled, so that the distribution characteristics of the matrix element values ​​in the second data matrix have the advantage of being easy to quantize. Accordingly, the quantization result of the model data obtained after quantization according to the second data matrix is ​​easier to compress, thereby helping to reduce the data volume of the model data, reduce the occupancy of the transmission bandwidth and the consumption of computing resources, and improve the processing efficiency of the model.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0026] Figure 1 A schematic diagram of a flow chart of a method for quantifying model data provided by an embodiment of the present disclosure is shown;

[0027] Figure 2 A schematic flow chart of a data processing method based on a neural network provided by another embodiment of the present disclosure is shown;

[0028] Figure 3 A block diagram of a device for quantizing model data provided by an embodiment of the present disclosure;

[0029] Figure 4 A block diagram of a data processing device based on a neural network provided in an embodiment of the present disclosure;

[0030] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0033] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0036] Figure 1 FIG. 1 is a flow chart showing a method for quantifying model data provided by an embodiment of the present disclosure. Figure 1 As shown, the quantification method includes:

[0037] Step S110: Acquire a first data matrix corresponding to the model data.

[0038] Among them, the model data in the embodiments of the present disclosure can be various types of data that need to be transmitted by the model during the training stage and / or the inference stage, including but not limited to: weight data, activation value data, etc.

[0039] The matrix size of the first data matrix corresponding to the model data can be determined according to the model structure. For example, taking the neural network model as an example, assuming that the model data needs to be transmitted from the first network layer of the neural network to the second network layer, the number of elements of the first data matrix in the first direction can be determined according to the number of neurons included in the first network layer of the neural network, and the number of elements of the first data matrix in the second direction can be determined according to the number of neurons included in the second network layer of the neural network.

[0040] It can be seen that by representing the model data transmitted between the various layers of the model in matrix form, it is helpful to achieve subsequent calculations and transmissions. This application does not limit the specific type of model data and the specific form of the first data matrix.

[0041] Step S120: Generate a scaling matrix corresponding to the first data matrix according to preset constraints, and scale the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraints are used to constrain the distribution characteristics of the matrix element values ​​in the second data matrix.

[0042] Wherein, the preset constraint condition is used to enable the scaling matrix to better control the distribution characteristics of the second data matrix after scaling. For example, the preset constraint condition is used to constrain the distribution characteristics of the matrix element values ​​in the second data matrix so that the distribution of the matrix element values ​​in the second data matrix is ​​more concentrated. By improving the concentration degree of the matrix element values ​​in the second data matrix, it is easy to implement subsequent compression processing. Those skilled in the art can flexibly set various types of constraint conditions, as long as the concentration degree of the matrix element values ​​in the second data matrix can be improved, and the present disclosure does not limit the specific types of constraint conditions.

[0043] Among them, when the first data matrix is ​​scaled according to the scaling matrix to obtain the second data matrix, the scaling of the first data matrix can be achieved by an element-by-element operation. For example, for each matrix element in the first data matrix, a preset operation (such as a division operation, a difference operation, etc.) can be performed using the corresponding element in the scaling matrix to obtain the element value of the corresponding element in the second data matrix. The present disclosure does not limit the specific operation type of the scaling process, and those skilled in the art can flexibly set it.

[0044] Step S130: obtaining a quantization result of the model data according to the second data matrix.

[0045] Since the concentration of matrix element values ​​in the second data matrix is ​​higher, it is convenient to represent more information content through a smaller quantization bit width. Therefore, the scaled second data matrix is ​​more convenient for quantization processing, which helps to reduce the amount of data in the quantization result. Among them, the present disclosure does not limit the specific implementation method for quantizing the second data matrix, and those skilled in the art can flexibly adopt various quantization methods.

[0046] In the embodiments provided in the present disclosure, the model data can be represented by a first data matrix. In order to reduce the data level of the first data matrix, in the present disclosure, a scaling matrix corresponding to the first data matrix is ​​generated according to preset constraints, and a second data matrix after scaling the first data matrix is ​​obtained according to the scaling matrix. Since the scaling matrix is ​​generated according to the preset constraints, the distribution characteristics of each matrix element value in the second data matrix obtained after scaling can be better controlled, so that the distribution characteristics of the matrix element values ​​in the second data matrix have the advantage of being easy to quantize. Accordingly, the quantization result of the model data obtained after quantization according to the second data matrix is ​​easier to compress, thereby helping to reduce the data volume of the model data, reduce the occupancy of the transmission bandwidth and the consumption of computing resources, and improve the processing efficiency of the model.

[0047] In addition, those skilled in the art may also make various modifications and variations to the embodiments of the present disclosure:

[0048] In an optional implementation, in order to achieve a better constraint effect, the preset constraint condition may include: a first constraint condition (also called a discrete degree constraint condition) for constraining the discrete degree of the matrix element values ​​in the second data matrix; and / or a second constraint condition (also called an information entropy constraint condition) for constraining the information entropy of the matrix element values ​​in the second data matrix. The discrete degree can be characterized by various mathematical indicators such as variance and / or standard deviation. The smaller the discrete degree, the higher the concentration of the matrix element values ​​in the second data matrix. Therefore, the first constraint condition may be: the discrete degree of the matrix element values ​​in the second data matrix is ​​the smallest. Similarly, the smaller the information entropy, the higher the concentration of the matrix element values ​​in the second data matrix. Therefore, the second constraint condition may be: the information entropy of the matrix element values ​​in the second data matrix is ​​the smallest.

[0049] It can be seen that the first constraint and the second constraint are used to measure the distribution characteristics of matrix element values ​​from different angles. Among them, the first constraint focuses on measuring the discreteness and stability of matrix element values ​​through indicators such as variance and standard deviation, while the second constraint focuses on measuring the uncertainty and complexity of information through indicators such as information entropy. Among them, the first constraint and the second constraint can be used alone or in combination. By using the two in combination, the distribution of matrix element values ​​can be better constrained.

[0050] In an optional implementation, in order to reduce the amount of data transmission and shorten the processing delay, the scaling matrix can be a low-rank matrix. Accordingly, when generating the scaling matrix corresponding to the first data matrix according to the preset constraints, it can be implemented in the following way: according to the matrix size of the first data matrix, determine the rank of the scaling matrix; according to the rank of the scaling matrix, generate the initial matrix; according to the preset optimization target, iteratively process the initial matrix to obtain the scaling matrix corresponding to the first data matrix; wherein the preset optimization target is determined according to the preset constraints. Among them, the low-rank matrix has a lower cost in the storage and transmission process, so using a low-rank matrix as a scaling matrix helps to improve processing efficiency. This method can transform the problem of generating a scaling matrix into an optimization target solving problem, thereby constructing an optimization objective function, and obtaining the scaling matrix by automatic learning. Since the optimization objective function aims to minimize the discrete degree or information entropy of the matrix element values ​​in the second data matrix, it is possible to solve the scaling matrix that meets the preset constraints.

[0051] The rank of the initial matrix matches the rank of the scaling matrix determined according to the matrix size of the first data matrix, for example, the rank of the initial matrix is ​​equal to the rank of the scaling matrix determined according to the matrix size of the first data matrix. The initial matrix can be determined in a variety of ways, for example, by random initialization or singular value decomposition.

[0052] In an optional implementation, the initial matrix and the scaling matrix are determined by singular value decomposition. Accordingly, when generating the initial matrix according to the rank of the scaling matrix, it can be obtained in the following manner: performing singular value decomposition on the first data matrix, extracting a preset number of singular values ​​that match the rank of the scaling matrix from the obtained multiple singular values; generating a first low-rank matrix and a second low-rank matrix according to the preset number of singular values; obtaining the initial matrix according to the product between the transposed matrix of the first low-rank matrix and the second low-rank matrix. Among them, the value of the preset number is equal to the value of the rank of the scaling matrix. The preset number is usually related to this method, and the initial matrix can be obtained by means of singular value decomposition, so that the assignment of the initial matrix is ​​more reasonable, thereby improving the generation efficiency of the scaling matrix.

[0053] In addition, the inventors found in the process of implementing the present invention that the numerical value of the rank of the scaling matrix will have a significant impact on the subsequent quantization operation: if the rank of the scaling matrix is ​​set too low (for example, the rank is 1), the regulating effect of the scaling matrix on the values ​​of each matrix element in the scaled second data matrix will be reduced, and the distribution of the values ​​of each matrix element in the second data matrix cannot be better controlled, resulting in poor quantization effect; conversely, if the rank of the scaling matrix is ​​set too high (for example, close to the full rank), the data volume of the scaling matrix itself will be large, and then a large bandwidth will be occupied in the subsequent transmission process. Therefore, the value of the rank of the scaling matrix is ​​related to the subsequent data processing efficiency. In an optional implementation method, the rank of the scaling matrix can be determined in the following way:

[0054] First, according to the matrix size of the first data matrix, the range of values ​​of the rank of the scaling matrix is ​​determined. The lower limit value in the range of values ​​of the rank of the scaling matrix is ​​greater than 1, and the upper limit value is less than a preset ratio of the matrix size of the first data matrix. For example, assuming that the matrix size of the first data matrix is ​​M*N, and N is less than or equal to M, the upper limit value of the rank of the scaling matrix is ​​less than a preset ratio of N. The preset ratio can be, for example, one tenth, one hundredth, or other values ​​less than 1.

[0055] Then, multiple candidate values ​​are set according to the above value range. For each candidate value, a scaling candidate matrix whose rank matches the candidate value is generated, thereby obtaining multiple scaling candidate matrices and multiple second candidate data matrices corresponding to the multiple scaling candidate matrices.

[0056] Finally, for each second candidate data matrix, the distribution characteristics of the matrix element values ​​in the second candidate data matrix are determined, and the rank of the scaling candidate matrix corresponding to the second candidate data matrix whose distribution characteristics meet the preset conditions is determined as the rank of the above-mentioned scaling matrix. For example, for each second candidate data matrix, the discreteness and / or information entropy of the matrix element values ​​in the second candidate data matrix are determined, and the rank of the scaling candidate matrix corresponding to a second candidate data matrix with the smallest discreteness and / or the smallest information entropy is determined as the rank of the above-mentioned scaling matrix. Accordingly, a second candidate data matrix with the smallest discreteness and / or the smallest information entropy is used as the second data matrix finally used for quantization. Among them, the distribution characteristics and preset conditions can also be flexibly characterized by various other methods, and the present disclosure does not limit this.

[0057] It can be seen that in the above method, multiple candidate values ​​can be generated within the specified value range, and corresponding scaling candidate matrices and second candidate data matrices are generated for each candidate, so as to determine the rank of the scaling matrix according to the second candidate data matrix with the best distribution effect. Through the above method, the number of ranks of the scaling matrix can be made more reasonable, thereby taking into account the efficiency of data transmission and the effect of quantization.

[0058] In addition, the value of the rank of the above-mentioned scaling matrix can be either a fixed value or a dynamically changing value. For example, in an optional implementation, considering that the model data between the layers within the same network model have certain commonalities, the rank of the scaling matrix corresponding to the first data matrix can be determined by the above-mentioned method during the initial training process of the network model, the first reasoning process, or the process of the first network layer transmitting the model data, and the rank of the scaling matrix is ​​fixedly used for calculation in the subsequent training process of the network model, the subsequent reasoning process, or the subsequent network layer transmitting the model data. This method does not need to calculate the rank of the scaling matrix for each model data to be transmitted, which helps to improve processing efficiency. In another optional implementation, the rank of the scaling matrix corresponding to the first data matrix can also be dynamically determined by the above-mentioned method during each training process of the network model, each reasoning process, or each network layer transmitting the model data. This method can make the rank of the scaling matrix determined each time more consistent with the characteristics of the current model data through dynamic calculation, which helps to improve the quantization effect.

[0059] In another optional implementation, the scaling matrix can also be a sparse matrix. Accordingly, when generating the scaling matrix corresponding to the first data matrix according to the preset constraints, it can be implemented in the following manner: generating a non-sparse matrix corresponding to the first data matrix according to the preset constraints; performing sparse processing on the non-sparse matrix according to the absolute values ​​of the matrix elements in the non-sparse matrix, and using the obtained sparse matrix as the scaling matrix corresponding to the first data matrix. For example, the absolute values ​​of each matrix element in the non-sparse matrix can be calculated, the element values ​​of several matrix elements with the largest absolute values ​​are retained, and the element values ​​of the remaining matrix elements are set to 0, thereby obtaining a sparse scaling matrix.

[0060] It can be seen that the generation method of the scaling matrix in the form of a sparse matrix is ​​similar to that of the scaling matrix in the form of a low-rank matrix, and both can be generated by iterative solution under the constraints of the objective function. Among them, the objective function can be generated according to the preset constraints mentioned above. The main difference between the two is that the scaling matrix in the form of a low-rank matrix is ​​further constrained by the low-rank condition in the process of solving according to the constraints of the objective function; the scaling matrix in the form of a sparse matrix is ​​further constrained by the sparse condition in the process of solving according to the constraints of the objective function. The present application does not limit the specific generation method of the scaling matrix in the form of a sparse matrix and the scaling matrix in the form of a low-rank matrix, and those skilled in the art can flexibly adopt various methods to achieve it.

[0061] It can be seen that the scaling matrix in the embodiment of the present disclosure can be flexibly implemented in various forms. In actual situations, in addition to low-rank matrices and sparse matrices, the scaling matrix can also be a matrix compressed by discrete cosine transform and other various other compressible matrix representation forms.

[0062] In an optional implementation, in order to further reduce the amount of data, when the quantization result of the model data is obtained according to the second data matrix, it can be implemented in the following way: according to the first bit width, the second data matrix is ​​quantized to obtain an initial quantization result matching the first bit width; the initial quantization result is subjected to bit width reduction processing to obtain a target quantization result matching the second bit width; wherein the second bit width is smaller than the first bit width. Wherein, the first bit width is the quantization bit width, which is used to characterize the number of bits used for the quantized model data (i.e., the initial quantization result). After obtaining the initial quantization result, it is further determined whether the initial quantization result meets the requirements of bit width reduction. If it meets the requirements, the initial quantization result is further subjected to bit width reduction processing so that the second bit width of the target quantization result finally obtained is smaller than the preset quantization bit width. For example, assuming that the preset quantization bit width (i.e., the first bit width) is 4 bits, if the bit widths of the various numerical values ​​in the initial quantization result are concentrated within the representation range of 3 bits, it is determined that the initial quantization result meets the requirements of bit width reduction, and the initial quantization result is further reduced to 3 bits. It can be seen that after the scaling adjustment of the scaling matrix, the distribution of the matrix element values ​​in the second data matrix is ​​more concentrated, so it is convenient to further perform bit width reduction processing on the quantization result to minimize the data occupancy of the final quantization result.

[0063] In an optional implementation, the model data may include: weight data in a neural network, and / or activation value data. The matrix size of the scaling matrix is ​​the same as the matrix size of the first data matrix. Accordingly, according to the scaling matrix, the second data matrix is ​​obtained by element-by-element operation. For example, the second data matrix may be generated in the following manner: for any first matrix element in the first data matrix, determine the scaling matrix element corresponding to the first matrix element in the scaling matrix; perform a preset operation on the first matrix element and the scaling matrix element, and determine the element value of the second matrix element corresponding to the first matrix element in the second data matrix according to the operation result. The preset operation may be a division operation, a subtraction operation, a mean operation, or other types of operations.

[0064] For example, for the first matrix element located at the mth row and nth column in the first data matrix, the element located at the mth row and nth column in the scaling matrix (i.e., the element at the same position) is used as the scaling matrix element corresponding to the first matrix element. Then, the element value of the first matrix element is divided by the element value of the scaling matrix element to obtain the element value of the second matrix element located at the mth row and nth column in the second data matrix. Wherein, m and n are natural numbers, and their value ranges are determined according to the matrix size.

[0065] In addition, another embodiment of the present disclosure also provides a data processing method based on a neural network. Figure 2 A flow chart of the data processing method is shown in FIG. Figure 2 As shown, the data processing method includes:

[0066] Step S210: Obtain model data corresponding to the first network layer of the neural network.

[0067] The first network layer of the neural network can be any network layer in the neural network. The model data corresponding to the first network layer can include: weight data, parameter data, activation value data of each neuron in the first network layer and other data that need to be transmitted to other network layers.

[0068] Step S220: Obtain the quantization result of the model data according to the quantization method of the model data.

[0069] Among them, it can be Figure 1 The quantization method shown is used to process the data, thereby obtaining the quantization result of the model data. For details, please refer to the relevant description in the previous embodiment, which will not be repeated here.

[0070] Step S230: Transmit the quantization result of the model data and the scaling matrix to the second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

[0071] The scaling matrix is ​​used to restore the original model data. The generation method thereof refers to the relevant description in the previous embodiment and will not be repeated here.

[0072] The second network layer of the neural network can also be any network layer in the neural network. For example, in the forward propagation scenario, the second network layer can be the next network layer of the first network layer; in the reverse propagation scenario, the second network layer can be the previous network layer of the first network layer.

[0073] Among them, the above-mentioned neural network-based data processing method can be applied to both the training process of the neural network and the reasoning process of the neural network. This application does not limit the specific scenario.

[0074] In an optional implementation, the second network layer can perform data processing in the following manner: the second network layer performs inverse quantization processing on the quantization result of the model data to obtain a second data matrix; performs inverse scaling processing on the second data matrix according to the scaling matrix to obtain the first data matrix, and performs data processing according to the first data matrix. It can be seen that the processing process of the second network layer is the inverse process of the processing process of the first network layer. Among them, the inverse quantization processing is the inverse process of the quantization processing, and the specific processing method depends on the selected quantization method. The inverse scaling processing is the inverse process of the scaling processing, and the specific processing method depends on the specific operation method of the scaling processing. For example, when the scaling processing adopts the operation method of element-by-element division, the inverse scaling processing can adopt the operation method of element-by-element multiplication, that is, the Hadamard product operation.

[0075] Through the above method, the distribution of the scaled matrix data can be better controlled with the help of the scaling matrix, thereby improving the quantization effect, significantly reducing the amount of data in the transmission process, and improving the model processing efficiency.

[0076] For ease of understanding, an example is used below to introduce the implementation of the method for quantizing model data provided by an embodiment of the present disclosure:

[0077] Large models (LM), especially large language models (LLM) such as GPT-3 (Generative Pre-trained Transformer-3) and GLM (General Language Model), have a huge number of parameters exceeding 100 billion, which brings huge challenges to deployment and application. At the same time, with the growing demand for privacy protection and timeliness, the necessity of local deployment of LLM on terminal devices such as mobile phones and laptops is increasing. Therefore, higher requirements are put forward for further reducing the storage space of LM and the transmission bandwidth during operation. In the reasoning scenario of large language models, the amount of calculation for each token generated is much smaller than the transmission amount of weight parameters. Therefore, compressing the total storage capacity or bandwidth transmission of large language models can greatly improve the reasoning speed.

[0078] Among them, weight compression is an effective way to solve the above problems. Weight compression mainly involves model quantization, model pruning, low-rank decomposition and other means. For example, model quantization refers to: reducing the amplitude representation accuracy of the model weight, compressing the 16-bit floating point number (fp16) of the weight into an 8-bit integer (int8) or a 4-bit integer (int4) or a lower bit width representation. Compared with the fp16 model performance, int4 precision quantization did not bring a significant decline. For another example, model pruning refers to: discarding unimportant weights in the model parameters to achieve sparsity. Low-rank decomposition refers to: using the low-rank characteristics of the weight matrix, discarding some of the singular values ​​and singular vectors with less influence, and decomposing the large weight matrix into the product of small weight matrices, thereby reducing the parameter storage. Among them, both model pruning and low-rank decomposition will significantly reduce the model accuracy, and therefore have a greater impact on the accuracy of the model. Therefore, model quantization is usually adopted.

[0079] In related technologies, the model quantization process can be expressed by the following formula:

[0080]

[0081] Among them, W, are the weight matrices before and after quantization, respectively. The weight matrix is ​​an implementation form of the first data matrix mentioned above. s is the scaling factor, which can generally take the maximum value of the absolute value of W. n is the quantization bit width. Accordingly, the dequantization process can be expressed by the following formula:

[0082]

[0083] The above-mentioned quantization method in the related art has at least the following defects: since the scaling factor s is generally a number or a vector, that is, quantized by row or column, the scaling factor has limited regulating effect on the element values ​​in the weight matrix (only the same adjustment process can be performed on all element values), and the differences between different element values ​​cannot be considered, resulting in unreasonable distribution of quantized data values ​​(for example, the distribution is not uniform enough), which in turn causes the average amount of information of the weight element to be significantly smaller than its bit width, wasting storage and bandwidth. In other words, the distribution of quantized data values ​​is not highly concentrated, the degree of discreteness is large, and the information entropy is large, resulting in the average amount of information of the quantized matrix being much smaller than the amount of information that can be represented by its bit width. Although variable-length technologies such as entropy coding can alleviate this problem, the serial decoding nature of entropy decoding greatly limits the decoding speed and its application.

[0084] In order to solve the above problems, this example generalizes the scaling factor s in the quantization method to a scaling matrix. Since the scaling matrix contains more element values, the control ability of the quantized data characteristics (such as distribution) is improved, while ensuring the numerical accuracy of the quantization, the subsequent compression effect is improved. The key to this example is: the scaling factor s in the quantization formula is generalized to a low-rank matrix with the same size as the original weight matrix. At this time, the matrix division operation (for ease of representation, this example takes division as an example) can be the element-by-element division of the two matrices. Among them, since the scaling factor is converted into a scaling matrix, the control ability of each element in the weight matrix becomes stronger. And because it is a low-rank matrix (for example, the rank is 2), the storage occupied by the scaling matrix is ​​much smaller than the data volume of the original weight matrix, thereby achieving the purpose of saving transmission bandwidth.

[0085] Optionally, the quantization formula in this example can be expressed as follows:

[0086]

[0087] Among them, U T V is a low-rank matrix of the same size as W. The above operations are all element-by-element operations on the matrix (such as element-by-element division). T V is used to represent the scaling matrix mentioned above. Since it is a division, the scaling matrix U T V can also be called a low-rank normalization matrix. It can be seen that the above formula is used to implement the scaling process and quantization process mentioned above. Among them, the weight matrix W corresponds to the first data matrix mentioned above. The result corresponds to the second data matrix mentioned above. Corresponding to the quantitative results mentioned above.

[0088] Accordingly, the above-mentioned inverse scaling process and inverse quantization process can be expressed by the following formula:

[0089]

[0090] The above operations are also element-by-element operations of matrix elements, such as element-by-element multiplication, i.e., Hadamard product. are the weight matrices before and after quantization respectively. n is the quantization bit width.

[0091] The low-rank normalization matrix (i.e., scaling matrix) can be determined by at least one of the following two methods:

[0092] (1) Singular value decomposition method: The original weight matrix is ​​subjected to singular value decomposition, and the largest part of singular values ​​(for example, 1 or 2) and the corresponding singular vectors are taken to form a scaling matrix.

[0093] (2) Automatic learning method: Given an optimization objective function, such as the distribution characteristics of the elements after constraint quantization (such as the variance of the second data matrix mentioned above, information entropy, etc.), the parameterized U is optimized by gradient descent, random search, etc. T V.

[0094] The above two methods can be used alone or in combination. In this example, in order to obtain better results, the above two methods are used in combination: the automatic learning method is mainly used for implementation, and the initial value required in the automatic learning method is determined by the singular value decomposition method.

[0095] In a specific example, the optimization objective function in the automatic learning method can be set to: minimize the variance or standard deviation of each matrix element value in the second data matrix obtained after scaling. The second data matrix is ​​the matrix obtained by dividing the weight matrix by the scaling matrix. In addition, the variance or standard deviation of each matrix element value in the second data matrix refers to: the normalized variance or standard deviation. The variance or standard deviation can be calculated after normalizing each matrix element value in the second data matrix according to its maximum absolute value to ensure that the absolute size of the variance or standard deviation can accurately indicate the degree of concentration of the data distribution.

[0096] For example, assuming that the original weight matrix is ​​8*8 in size, the specific data is as follows:

[0097] W=

[0098] [[0.1359,-0.1379,-0.3070,-0.1670,-0.5810,-0.2427,-0.1452,0.1877],

[0099] [-0.0638,-0.1836,-1.0000,0.2582,-0.0423,-0.0918,0.0307,0.2573],

[0100] [-0.2738,0.0902,0.3298,0.1328,0.1847,0.1690,-0.5393,-0.3133],

[0101] [0.0664,-0.0187,-0.0928,0.4233,-0.1629,-0.1390,-0.3258,0.3423],

[0102] [-0.5475,0.4294,-0.2386,-0.6017,-0.1970,0.3258,-0.0793,-0.1711],

[0103] [-0.5250,-1.0000,-0.7801,0.4357,-0.2022,-0.5890,-0.1856,-0.4254],

[0104] [-0.0518,-0.3237,-0.6431,-0.1681,0.1042,0.1742,0.0892,0.3444],

[0105] [0.2199,-0.3837,-0.7674,-0.5518,0.3983,0.2780,-0.0985,-0.0996]]

[0106] The standard deviation obtained after normalization of the weight matrix is ​​0.3502, which reflects the concentration of the data. The more concentrated the data is, the smaller the standard deviation is, which facilitates further bit width compression processing (ie, bit width reduction processing) in the subsequent processing process.

[0107] Among them, the automatic learning method can be used to determine U T V, assuming the rank is 2, then U T The size of is 8*2, the size of V is 2*8, and their values ​​are initialized by various methods such as Gaussian distribution. Then the standard deviation of the weight matrix (i.e., the second data matrix) after low-rank normalization is calculated as the objective function, and gradient descent is applied for iterative solution. After 10 iterations,

[0108] U T =

[0109] [[0.2500,0.0938],

[0110] [1.3140,1.9981],

[0111] [-0.3705,2.2845],

[0112] [-0.2989,0.6012],

[0113] [-0.0168,-0.5043],

[0114] [0.6487,-2.1925],

[0115] [-0.4963,-1.3197],

[0116] [-0.2323,0.1399]]

[0117] V=

[0118] [[-0.8569,0.3590,-0.5812,-0.0919,2.0623,0.0825,1.4824,-0.4686],

[0119] [-2.2452,-0.2629,-0.4807,-1.5078,0.6564,-1.6301,-0.3658,0.2207]]

[0120] U T V=

[0121] [[-0.4248,0.0651,-0.1904,-0.1644,0.5772,-0.1323,0.3363,-0.0965],

[0122] [-5.6122,-0.0535,-1.7243,-3.1335,4.0215,-3.1487,1.2170,-0.1748],

[0123] [-4.8116,-0.7335,-0.8829,-3.4105,0.7356,-3.7545,-1.3850,0.6777],

[0124] [-1.0937,-0.2654,-0.1153,-0.8791,-0.2218,-1.0048,-0.6631,0.2727],

[0125] [1.1465,0.1265,0.2522,0.7619,-0.3657,0.8206,0.1596,-0.1034],

[0126] [4.3667,0.8092,0.6770,3.2463,-0.1015,3.6276,1.7638,-0.7877],

[0127] [3.3882,0.1687,0.9229,2.0354,-1.8899,2.1102,-0.2530,-0.0586],

[0128] [-0.1151,-0.1202,0.0677,-0.1896,-0.3872,-0.2473,-0.3955,0.1397]]

[0129] Accordingly,

[0130]

[0131]

[0132] The "+0.0000001" added in the formula is to prevent division by 0 and improve the stability of numerical calculation. The matrix obtained after the above division operation is the second data matrix. The standard deviation of the second data matrix calculated after normalization is 0.1773, which is 50% lower than 0.3502, which means that after the data is processed by the low-rank normalization of the scaling matrix, the concentration of data distribution is effectively improved, providing space for subsequent bit width reduction.

[0133] In addition, this example can also be flexibly adapted to other types of quantization methods. For example, this example can also be applied to quantization methods such as GPTQ (Gradient-based Progressive Tensor Quantization) and AWQ (Activation-aware Weight Quantization), so as to expand the scaling coefficients in the above quantization methods into scaling matrices, thereby improving the quantization effect of the above quantization methods. In short, this application does not limit the specific implementation details of the quantization method. In any scenario where quantization can be performed by scaling coefficients, the solution in this example can be used to expand the scaling coefficients into scaling matrices to improve the quantization effect.

[0134] It can be seen that this example can improve uniformity by comprehensively fine-tuning the values ​​of matrix elements by scaling the matrix before quantization. In addition, this example is used to perform data uniformity processing in a quantized manner before data storage, and restore the original weight data through "dequantization" or "deuniformity processing" during the reading and use stage after storage. Among them, the specific implementation details of "dequantization" or "deuniformity processing" depend on the specific algorithms used for quantization and uniformity processing. For example, in this example, the formula mentioned above can be used for "dequantization" or "deuniformity processing".

[0135] The core of this example is to use as few parameters as possible and a constrained representation (such as matrix low-rank constraints, etc.) to calculate the scaling factor of each weight element to make the quantized data easier to compress (for example, more evenly distributed). This method can greatly improve the control ability of the quantized data characteristics while adding a small number of parameters, while ensuring accuracy and reducing the storage and communication requirements of the quantized data.

[0136] Of course, this example is not limited to low-rank normalized matrices, and other compressible matrix representations can also be flexibly adopted, such as sparse representation matrices, discrete cosine transform compressed matrices, etc., and the present disclosure does not limit this.

[0137] In addition, the data processing method in this example can be applied to one or more processing cores in a multi-core system. It can be understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this disclosure will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0138] Figure 3 A block diagram of a device for quantizing model data provided by an embodiment of the present disclosure. Figure 3 , the embodiment of the present disclosure provides a device for quantizing model data, the device comprising:

[0139] A first acquisition module 31, adapted to acquire a first data matrix corresponding to the model data;

[0140] A generating module 32 is adapted to generate a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and scale the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values ​​in the second data matrix;

[0141] The first quantization module 33 is adapted to obtain a quantization result of the model data according to the second data matrix.

[0142] In an optional implementation, the preset constraint condition includes:

[0143] A first constraint condition for constraining the discreteness of matrix element values ​​in the second data matrix; and / or,

[0144] A second constraint condition for constraining the information entropy of matrix element values ​​in the second data matrix.

[0145] In an optional implementation, the first constraint includes: a discrete degree constraint for constraining the discrete degree of matrix element values ​​in the second data matrix; the second constraint includes: an information entropy constraint for constraining the information entropy of matrix element values ​​in the second data matrix; wherein the discrete degree is characterized by variance and / or standard deviation.

[0146] In an optional implementation, the scaling matrix is ​​a low-rank matrix, and the generating module 32 is used to:

[0147] Determining the rank of the scaling matrix according to the matrix size of the first data matrix;

[0148] Generate an initial matrix according to the rank of the scaling matrix;

[0149] According to a preset optimization target, the initial matrix is ​​iteratively processed to obtain a scaling matrix corresponding to the first data matrix; wherein the preset optimization target is determined according to the preset constraint condition.

[0150] In an optional implementation, the generating module 32 is used to:

[0151] Performing singular value decomposition processing on the first data matrix, and extracting a preset number of singular values ​​matching the rank of the scaling matrix from the obtained multiple singular values;

[0152] Generate a first low-rank matrix and a second low-rank matrix according to the preset number of singular values;

[0153] The initial matrix is ​​obtained according to the product of the transposed matrix of the first low-rank matrix and the second low-rank matrix.

[0154] In an optional implementation, the scaling matrix is ​​a sparse matrix, and the generating module 32 is used to:

[0155] According to preset constraints, a non-sparse matrix corresponding to the first data matrix is ​​generated;

[0156] The non-sparse matrix is ​​sparsely processed according to the absolute values ​​of the matrix elements in the non-sparse matrix, and the obtained sparse matrix is ​​used as a scaling matrix corresponding to the first data matrix.

[0157] In an optional implementation, the first quantization module 33 is used to:

[0158] According to the first bit width, quantizing the second data matrix to obtain an initial quantization result matching the first bit width;

[0159] The initial quantization result is subjected to bit width reduction processing to obtain a target quantization result matching a second bit width; wherein the second bit width is smaller than the first bit width.

[0160] In an optional implementation, the model data includes: weight data and / or activation value data in a neural network;

[0161] The matrix size of the scaling matrix is ​​the same as the matrix size of the first data matrix, and the generating module 32 is used for:

[0162] For any first matrix element in the first data matrix, determining a scaling matrix element in the scaling matrix corresponding to the first matrix element;

[0163] A preset operation is performed on the first matrix element and the scaling matrix element, and an element value of a second matrix element corresponding to the first matrix element in the second data matrix is ​​determined according to the operation result.

[0164] Figure 4 A block diagram of a data processing device based on a neural network provided by an embodiment of the present disclosure. Figure 4 , the data processing device of the embodiment of the present disclosure includes:

[0165] A second acquisition module 41 is adapted to acquire model data corresponding to the first network layer of the neural network;

[0166] A second quantization module 42, adapted to obtain a quantization result of the model data according to any one of the quantization methods of claims 1-7;

[0167] The transmission module 43 is suitable for transmitting the quantization result and the scaling matrix of the model data to the second network layer of the neural network, so that the second network layer performs data processing according to the quantization result and the scaling matrix of the model data.

[0168] In an optional implementation, the second network layer performs data processing according to the quantization results of the model data and the scaling matrix, including: the second network layer performs inverse quantization processing on the quantization results of the model data to obtain a second data matrix; performs inverse scaling processing on the second data matrix according to the scaling matrix to obtain the first data matrix, and performs data processing according to the first data matrix.

[0169] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0170] Reference Figure 5 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 901; at least one memory 902, and one or more I / O interfaces 903, connected between the processor 901 and the memory 902; wherein the memory 902 stores one or more computer programs that can be executed by at least one processor 901, and the one or more computer programs are executed by at least one processor 901 so that the at least one processor 901 can execute the above-mentioned model training method.

[0171] In some embodiments, the processing device may be a brain-like chip, which can use vectorized computing and needs to load the weight information and other parameters of the neural network model through an external memory such as a double data rate (DDR) synchronous dynamic random access memory. Therefore, the batch processing used in the embodiments of the present disclosure has a higher computing efficiency.

[0172] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned model training method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0173] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0174] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0175] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0176] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0177] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0178] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0179] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0180] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0181] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0182] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0183] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A method for quantifying model data, the method comprising: Acquire a first data matrix corresponding to the model data; According to preset constraints, a scaling matrix corresponding to the first data matrix is ​​generated, and the first data matrix is ​​scaled according to the scaling matrix to obtain a second data matrix; wherein the preset constraints are used to constrain the distribution characteristics of matrix element values ​​in the second data matrix; A quantization result of the model data is obtained according to the second data matrix.

2. The method according to claim 1, wherein: The preset constraints include: A first constraint condition for constraining the discreteness of matrix element values ​​in the second data matrix; and / or, A second constraint condition for constraining the information entropy of matrix element values ​​in the second data matrix.

3. The method according to claim 2, wherein: The first constraint condition includes: a discrete degree constraint condition for constraining the discrete degree of matrix element values ​​in the second data matrix; the second constraint condition includes: an information entropy constraint condition for constraining the information entropy of matrix element values ​​in the second data matrix; wherein the discrete degree is characterized by variance and / or standard deviation.

4. The method according to any one of claims 1 to 3, wherein: The scaling matrix is ​​a low-rank matrix, and generating the scaling matrix corresponding to the first data matrix according to the preset constraint condition includes: Determining the rank of the scaling matrix according to the matrix size of the first data matrix; Generate an initial matrix according to the rank of the scaling matrix; According to a preset optimization target, the initial matrix is ​​iteratively processed to obtain a scaling matrix corresponding to the first data matrix; wherein the preset optimization target is determined according to the preset constraint condition.

5. The method according to claim 4, wherein: Generating an initial matrix according to the rank of the scaling matrix comprises: Performing singular value decomposition processing on the first data matrix, and extracting a preset number of singular values ​​matching the rank of the scaling matrix from the obtained multiple singular values; Generate a first low-rank matrix and a second low-rank matrix according to the preset number of singular values; The initial matrix is ​​obtained according to the product of the transposed matrix of the first low-rank matrix and the second low-rank matrix.

6. The method according to any one of claims 1 to 3, wherein: The scaling matrix is ​​a sparse matrix, and generating the scaling matrix corresponding to the first data matrix according to the preset constraint condition includes: According to preset constraints, a non-sparse matrix corresponding to the first data matrix is ​​generated; The non-sparse matrix is ​​sparsely processed according to the absolute values ​​of the matrix elements in the non-sparse matrix, and the obtained sparse matrix is ​​used as a scaling matrix corresponding to the first data matrix.

7. The method according to any one of claims 1 to 3, wherein: The obtaining of the quantization result of the model data according to the second data matrix comprises: According to the first bit width, quantizing the second data matrix to obtain an initial quantization result matching the first bit width; The initial quantization result is subjected to bit width reduction processing to obtain a target quantization result matching a second bit width; wherein the second bit width is smaller than the first bit width.

8. The method according to any one of claims 1 to 3, wherein: The model data includes: weight data and / or activation value data in a neural network; The matrix size of the scaling matrix is ​​the same as the matrix size of the first data matrix, and scaling the first data matrix according to the scaling matrix to obtain the second data matrix includes: For any first matrix element in the first data matrix, determining a scaling matrix element in the scaling matrix corresponding to the first matrix element; A preset operation is performed on the first matrix element and the scaling matrix element, and an element value of a second matrix element corresponding to the first matrix element in the second data matrix is ​​determined according to the operation result.

9. The method according to claim 8, wherein: The preset operation includes a division operation, and the quantization result of the model data is obtained by the following formula: in, The quantization result used to characterize the model data, W is used to characterize the first data matrix, U T V is used to represent the scaling matrix, and n is used to represent the quantization bit width during quantization processing.

10. A data processing method based on a neural network, comprising: Obtaining model data corresponding to the first network layer of the neural network; According to any one of claims 1 to 8, obtaining a quantization result of the model data; The quantization results and scaling matrix of the model data are transmitted to the second network layer of the neural network, so that the second network layer performs data processing according to the quantization results and scaling matrix of the model data.

11. The method according to claim 10, wherein: The second network layer performs data processing according to the quantization result of the model data and the scaling matrix, including: The second network layer performs inverse quantization processing on the quantization result of the model data to obtain a second data matrix; The second data matrix is ​​subjected to inverse scaling processing according to the scaling matrix to obtain the first data matrix, and data processing is performed according to the first data matrix.

12. The method according to claim 11, wherein: The first data matrix is ​​obtained by the following formula: in, The quantization result used to characterize the model data, W is used to characterize the first data matrix, U T V is used to represent the scaling matrix, and n is used to represent the inverse quantization bit width during inverse quantization processing.

13. A device for quantifying model data, the device comprising: A first acquisition module, adapted to acquire a first data matrix corresponding to the model data; A generating module, adapted to generate a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and scale the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values ​​in the second data matrix; The first quantization module is adapted to obtain a quantization result of the model data according to the second data matrix.

14. A data processing device based on a neural network, comprising: A second acquisition module, adapted to acquire model data corresponding to the first network layer of the neural network; A second quantization module, adapted to obtain a quantization result of the model data according to the quantization method according to any one of claims 1 to 8; The transmission module is suitable for transmitting the quantization results and scaling matrix of the model data to the second network layer of the neural network, so that the second network layer performs data processing according to the quantization results and scaling matrix of the model data.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 12.

Images