Model data quantization method and apparatus, and electronic device and readable storage medium

Abstract

Provided in the present disclosure are a model data quantization method and apparatus, and an electronic device and a readable storage medium. The model data quantization method comprises: acquiring a first data matrix corresponding to model data; on the basis of a preset constraint condition, generating a scaling matrix corresponding to the first data matrix, and on the basis of the scaling matrix, performing scaling processing on the first data matrix, so as to obtain a second data matrix, wherein the preset constraint condition is used for constraining a distribution feature of matrix element values in the second data matrix; and on the basis of the second data matrix, obtaining a quantization result of the model data. The method helps reduce the data volume of model data, and reduce the occupancy of the transmission bandwidth and the consumption of computing resources, thereby improving the processing efficiency of a model.

Description

Quantization method and device of model data, electronic equipment and readable storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to a quantization method and device of model data, an electronic equipment and a readable storage medium. BACKGROUND

[0002] Model data includes various types of data used to construct and train a model, as well as data required in the model inference process. Model data can be an abstract result of data features in a model, used to describe the static features, dynamic behavior and constraint conditions of a system, etc. As can be seen, model data plays a crucial role in the model training and inference process.

[0003] In an application scenario with a large amount of data, the order of magnitude of model data is usually large, so when transmitting model data within the model, it will cause a large amount of transmission bandwidth occupation and a large amount of computing resource consumption, thereby greatly affecting the processing efficiency of the model. As can be seen, how to reduce the data volume of model data has become a technical problem to be solved at present. SUMMARY

[0004] The present disclosure provides a quantization method and device of model data, an electronic equipment and a readable storage medium.

[0005] In a first aspect, the present disclosure provides a quantization method of model data, comprising:

[0006] obtaining a first data matrix corresponding to the model data;

[0007] generating a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and performing scaling processing on the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values in the second data matrix;

[0008] obtaining a quantization result of the model data according to the second data matrix.

[0009] In a second aspect, the present disclosure provides a data processing method based on a neural network, comprising:

[0010] obtaining model data corresponding to a first network layer of the neural network;

[0011] obtaining a quantization result of the model data according to the quantization method described above;

[0012] The quantization result of the model data and the scaling matrix are transmitted to a second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

[0013] In a third aspect, the present disclosure provides a model data quantization device, which comprises:

[0014] A first acquisition module is adapted to acquire a first data matrix corresponding to the model data.

[0015] A generation module is adapted to generate a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and to perform scaling processing on the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values in the second data matrix.

[0016] A first quantization module is adapted to obtain a quantization result of the model data according to the second data matrix.

[0017] In a fourth aspect, the present disclosure provides a neural network-based data processing device, which comprises:

[0018] A second acquisition module is adapted to acquire model data corresponding to a first network layer of the neural network.

[0019] A second quantization module is adapted to obtain a quantization result of the model data according to the above-mentioned quantization method.

[0020] A transmission module is adapted to transmit the quantization result of the model data and the scaling matrix to a second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

[0021] In a fifth aspect, the present disclosure provides an electronic device, which comprises at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0022] In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processing core, implements the above-mentioned method.

[0023] In a seventh aspect, the present disclosure provides a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying the computer readable code, wherein when the computer readable code is run in a processor of an electronic device, the processor in the electronic device performs the method described above.

[0024] In the embodiments provided by the present disclosure, the model data can be represented by a first data matrix, in order to reduce the data magnitude of the first data matrix, in the present disclosure, a scaling matrix corresponding to the first data matrix is generated according to a preset constraint condition, and a second data matrix obtained by scaling the first data matrix is obtained according to the scaling matrix. Since the scaling matrix is generated according to the preset constraint condition, the distribution characteristics of each matrix element value in the second data matrix obtained after scaling can be better controlled, so that the distribution characteristics of the matrix element values in the second data matrix have the advantage of being easy to quantify. Correspondingly, the quantization result of the model data obtained after quantization according to the second data matrix is more convenient for compression, thereby helping to reduce the data amount of the model data, reduce the occupation of the transmission bandwidth and the consumption of the computing resources, and improve the processing efficiency of the model.

[0025] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the embodiments of the present disclosure are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of the specific example embodiments, with reference to the accompanying drawings, in which:

[0027] FIG. 1 shows a flowchart of a model data quantization method according to an embodiment of the present disclosure;

[0028] FIG. 2 shows a flowchart of a neural network-based data processing method according to another embodiment of the present disclosure;

[0029] FIG. 3 is a block diagram of a model data quantization device according to an embodiment of the present disclosure;

[0030] FIG. 4 is a block diagram of a neural network-based data processing device according to an embodiment of the present disclosure;

[0031] FIG. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the drawings, including various details of the embodiments of the present disclosure to facilitate understanding. Those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures.

[0033] The embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0034] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0035] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "comprise" and / or "consist of", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "connected" or "coupled" and / or similar terms are not limited to a physical or mechanical connection or linkage to either another part or device and can also include an electrical connection, whether direct or indirect.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly formal or overly strict sense unless expressly so defined herein.

[0037] FIG. 1 shows a flowchart of a model data quantization method according to an embodiment of the present disclosure. As shown in FIG. 1, the quantization method includes:

[0038] Step S110: Obtain a first data matrix corresponding to model data.

[0039] In the present disclosure, the model data can be various types of data required for the model to be transmitted during the training phase and / or the inference phase, including but not limited to weight data, activation value data, etc.

[0040] The matrix size of the first data matrix corresponding to the model data can be determined according to the model structure. For example, taking a neural network model as an example, assuming that the model data needs to be transmitted from the first network layer of the neural network to the second network layer, the number of elements of the first data matrix in the first direction can be determined according to the number of neurons included in the first network layer of the neural network, and the number of elements of the first data matrix in the second direction can be determined according to the number of neurons included in the second network layer of the neural network.

[0041] Therefore, by representing the model data transmitted between the layer structures of the model as a matrix form, it is helpful to implement subsequent operations and transmissions. The present disclosure does not limit the specific types of model data and the specific forms of the first data matrix.

[0042] Step S120: generating a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and performing scaling processing on the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values in the second data matrix.

[0043] The preset constraint condition is used to enable the scaling matrix to better control the distribution characteristics of the scaled second data matrix. For example, the preset constraint condition is used to constrain the distribution characteristics of the matrix element values in the second data matrix, so that the distribution of the matrix element values in the second data matrix is more concentrated. By improving the concentration degree of the matrix element values in the second data matrix, subsequent compression processing can be facilitated. Those skilled in the art can flexibly set various types of constraint conditions, as long as the concentration degree of the matrix element values in the second data matrix is improved. The present disclosure does not limit the specific types of constraint conditions.

[0044] When the scaling processing on the first data matrix is performed according to the scaling matrix to obtain the second data matrix, the scaling of the first data matrix can be implemented by an element-by-element level operation mode. For example, for each matrix element in the first data matrix, a preset operation (such as division operation, difference operation, etc.) can be performed on the corresponding element in the scaling matrix to obtain the element value of the corresponding element in the second data matrix. The present disclosure does not limit the specific operation types of the scaling processing, which can be flexibly set by those skilled in the art.

[0045] Step S130: obtaining a quantization result of the model data according to the second data matrix.

[0046] Since the concentration of the matrix element values in the second data matrix is higher, more information content can be represented by fewer quantization bit widths. Therefore, the scaled second data matrix is more convenient for quantization processing, which helps to reduce the data amount of the quantization result. In the present disclosure, the specific implementation of quantizing the second data matrix is not limited, and various quantization methods can be flexibly used by those skilled in the art.

[0047] In the embodiments provided in the present disclosure, the model data can be represented by the first data matrix. In order to reduce the data amount of the first data matrix, in the present disclosure, a scaling matrix corresponding to the first data matrix is generated according to a preset constraint condition, and a second data matrix obtained by scaling the first data matrix is obtained according to the scaling matrix. Since the scaling matrix is generated according to the preset constraint condition, the distribution characteristics of each matrix element value in the second data matrix obtained after scaling can be better controlled, so that the distribution characteristics of the matrix element values in the second data matrix have the advantage of being convenient for quantization. Correspondingly, the quantization result of the model data obtained by quantizing the second data matrix is more convenient for compression, thereby helping to reduce the data amount of the model data, reduce the occupation of the transmission bandwidth and the consumption of the computing resources, and improve the processing efficiency of the model.

[0048] In addition, those skilled in the art can also make various modifications and deformations to the embodiments of the present disclosure:

[0049] In an optional implementation, in order to achieve a better constraint effect, the preset constraint condition can include: a first constraint condition (also called a dispersion degree constraint condition) for constraining the dispersion degree of the matrix element values in the second data matrix; and / or, a second constraint condition (also called an information entropy constraint condition) for constraining the information entropy of the matrix element values in the second data matrix. The dispersion degree can be represented by various mathematical indicators such as variance, and / or standard deviation. The smaller the dispersion degree, the higher the concentration of the matrix element values in the second data matrix. Therefore, the first constraint condition can be that the dispersion degree of the matrix element values in the second data matrix is the smallest. Similarly, the smaller the information entropy, the higher the concentration of the matrix element values in the second data matrix. Therefore, the second constraint condition can be that the information entropy of the matrix element values in the second data matrix is the smallest.

[0050] As can be seen, the first constraint condition and the second constraint condition are respectively used to measure the distribution characteristics of the matrix element values from different angles. The first constraint condition focuses on measuring the dispersion degree and stability of the matrix element values through indicators such as variance and standard deviation, while the second constraint condition focuses on measuring the uncertainty and complexity of the information through indicators such as information entropy. The first constraint condition and the second constraint condition can be used alone or in combination. By using the two in combination, the distribution of the matrix element values can be better constrained.

[0051] In an optional implementation, in order to reduce the data transmission amount and shorten the processing delay, the scaling matrix can be a low-rank matrix. Correspondingly, when the scaling matrix corresponding to the first data matrix is generated according to the preset constraint condition, the following manner can be used: determining the rank of the scaling matrix according to the matrix size of the first data matrix; generating an initial matrix according to the rank of the scaling matrix; and performing iterative processing on the initial matrix according to a preset optimization target to obtain the scaling matrix corresponding to the first data matrix; wherein the preset optimization target is determined according to the preset constraint condition. The low-rank matrix has a lower cost in storage and transmission, and therefore, using the low-rank matrix as the scaling matrix helps to improve the processing efficiency. This manner can convert the problem of generating the scaling matrix into an optimization target solving problem, so as to construct an optimization target function and obtain the scaling matrix through automatic learning. Since the optimization target function aims to minimize the dispersion degree of the matrix element values in the second data matrix or minimize the information entropy, the scaling matrix that satisfies the preset constraint condition can be solved.

[0052] The rank of the initial matrix matches the rank of the scaling matrix determined according to the matrix size of the first data matrix, for example, the rank of the initial matrix is equal to the rank of the scaling matrix determined according to the matrix size of the first data matrix. The initial matrix can be determined in various ways, for example, it can be determined through a random initialization manner or a singular value decomposition manner.

[0053] In an optional implementation, the initial matrix and the scaling matrix are determined through a singular value decomposition manner. Correspondingly, when the initial matrix is generated according to the rank of the scaling matrix, the following manner can be used: performing singular value decomposition processing on the first data matrix, extracting a preset number of singular values matching the rank of the scaling matrix from the obtained multiple singular values; generating a first low-rank matrix and a second low-rank matrix according to the preset number of singular values; and obtaining the initial matrix according to the product between the transpose matrix of the first low-rank matrix and the second low-rank matrix. The value of the preset number is equal to the value of the rank of the scaling matrix. The initial matrix is obtained through the singular value decomposition manner, so that the assignment of the initial matrix is more reasonable, and the generation efficiency of the scaling matrix is improved.

[0054] In addition, the inventor found in the process of implementing the present application that the numerical value of the rank of the scaling matrix has a significant impact on the subsequent quantization operation: if the rank of the scaling matrix is set too low (for example, rank 1), the adjustment effect of the scaling matrix on the values of the elements of the scaled second data matrix will be reduced, and the distribution of the values of the elements of the second data matrix cannot be better controlled, resulting in poor quantization effect; on the contrary, if the rank of the scaling matrix is set too high (for example, close to the full rank), the data amount of the scaling matrix itself will be large, and thus a large bandwidth will be occupied in the subsequent transmission process. Therefore, the value of the rank of the scaling matrix relates to the efficiency of subsequent data processing. In an optional implementation, the rank of the scaling matrix can be determined in the following manner:

[0055] First, according to the matrix size of the first data matrix, the value range of the rank of the scaling matrix is determined. The lower limit value in the value range of the rank of the scaling matrix is greater than 1, and the upper limit value is less than a preset proportion of the matrix size of the first data matrix. For example, assuming that the matrix size of the first data matrix is M*N, and N is less than or equal to M, the upper limit value of the rank of the scaling matrix is less than a preset proportion of N. The preset proportion may be, for example, one-tenth, or one-hundredth, and various other values less than 1.

[0056] Then, according to the above value range, a plurality of candidate values are set. For each candidate value, a scaling candidate matrix with a rank matching the candidate value is generated, thereby obtaining a plurality of scaling candidate matrices and a plurality of second candidate data matrices corresponding to the plurality of scaling candidate matrices.

[0057] Finally, for each second candidate data matrix, the distribution characteristics of the matrix element values in the second candidate data matrix are determined, and the rank of the scaling candidate matrix corresponding to the second candidate data matrix whose distribution characteristics satisfy the preset condition is determined as the rank of the above scaling matrix. For example, for each second candidate data matrix, the dispersion degree and / or information entropy of the matrix element values in the second candidate data matrix are determined, and the rank of the scaling candidate matrix corresponding to the second candidate data matrix with the smallest dispersion degree and / or information entropy is determined as the rank of the above scaling matrix. Correspondingly, the second candidate data matrix with the smallest dispersion degree and / or information entropy is taken as the final second data matrix for quantization. The distribution characteristics and the preset condition can also be flexibly represented by other various methods, which are not limited in the present disclosure.

[0058] Therefore, in the above manner, multiple candidate values can be generated within the specified value range, and a corresponding scaling candidate matrix and a second candidate data matrix are generated for each candidate, so as to determine the rank of the scaling matrix according to the second candidate data matrix with the best distribution effect. Through the above manner, the number of ranks of the scaling matrix can be more reasonable, so as to balance the efficiency of data transmission and the quantization effect.

[0059] In addition, the value of the rank of the scaling matrix can be a fixed value or a dynamically changed value. For example, in an optional implementation manner, considering that the model data between the layers in the same network model has certain commonality, the rank of the scaling matrix corresponding to the first data matrix can be determined by the above-mentioned manner in the initial training process of the network model, in the first inference process, or in the process of transmitting model data of the first network layer, and the rank of the scaling matrix is fixedly used for calculation in the subsequent training process of the network model, in the subsequent inference process, or in the process of transmitting model data of the subsequent network layer. This manner does not need to calculate the rank of the scaling matrix for each to-be-transmitted model data, thereby helping to improve the processing efficiency. In another optional implementation manner, the rank of the scaling matrix corresponding to the first data matrix can be dynamically determined by the above-mentioned manner in each training process of the network model, in each inference process, or in the process of transmitting model data of each network layer. Through dynamic calculation, this manner can make the rank of the scaling matrix determined each time more conform to the characteristics of the current model data, thereby helping to improve the quantization effect.

[0060] In another optional implementation manner, the scaling matrix can also be a sparse matrix. Correspondingly, when the scaling matrix corresponding to the first data matrix is generated according to the preset constraint condition, the following manner can be used: a non-sparse matrix corresponding to the first data matrix is generated according to the preset constraint condition; the non-sparse matrix is sparsified according to the absolute values of the matrix elements in the non-sparse matrix, and the obtained sparse matrix is taken as the scaling matrix corresponding to the first data matrix. For example, the absolute values of the matrix elements in the non-sparse matrix can be calculated, the element values of a plurality of matrix elements with the largest absolute values are retained, and the element values of the remaining matrix elements are all set to 0, so as to obtain the sparsified scaling matrix.

[0061] It can be seen that the generation manner of the scaling matrix in the form of sparse matrix is similar to the scaling matrix in the form of low-rank matrix, and both can be generated by iterative solving under the constraint of the objective function. The main difference between the two is that the scaling matrix in the form of low-rank matrix is further constrained by the low-rank condition in the solving process under the constraint of the objective function, and the scaling matrix in the form of sparse matrix is further constrained by the sparse condition in the solving process under the constraint of the objective function. The specific generation manner of the scaling matrix in the form of sparse matrix and the scaling matrix in the form of low-rank matrix is not limited in the disclosure, and various manners can be flexibly adopted by those skilled in the art.

[0062] It can be seen that the scaling matrix in the embodiments of the disclosure can take various implementation forms flexibly. In actual situations, in addition to low-rank matrix and sparse matrix, the scaling matrix can also be other various compressible matrix representations such as discrete cosine transform compressed matrix.

[0063] In an optional implementation, in order to further reduce the data amount, when obtaining the quantization result of the model data according to the second data matrix, the following manner can be adopted: quantizing the second data matrix according to the first bit width to obtain an initial quantization result matched with the first bit width; performing bit width reduction processing on the initial quantization result to obtain a target quantization result matched with the second bit width; wherein the second bit width is smaller than the first bit width. The first bit width is the quantization bit width, which is used to represent the number of bits used by the quantized model data (i.e. the initial quantization result). After obtaining the initial quantization result, it is further determined whether the initial quantization result meets the requirement of bit width reduction. If it meets the requirement, the initial quantization result is further processed by bit width reduction, so that the second bit width of the target quantization result obtained finally is smaller than the preset quantization bit width. For example, assuming that the preset quantization bit width (i.e. the first bit width) is 4 bits, if the bit width of each value in the initial quantization result is concentrated in the range of 3 bits, it is determined that the initial quantization result meets the requirement of bit width reduction, and the initial quantization result is further reduced to 3 bits. It can be seen that, since the distribution of the matrix element values in the second data matrix is more concentrated after the scaling adjustment by the scaling matrix, it is convenient to further perform bit width reduction processing on the quantization result to reduce the data amount of the quantization result obtained finally to the greatest extent.

[0064] In an optional implementation, the model data can include weight data in the neural network and / or activation value data. The matrix size of the scaling matrix is the same as the matrix size of the first data matrix. According to the scaling matrix, the second data matrix is obtained through element-by-element operation. For example, the second data matrix can be generated in the following manner: for any first matrix element in the first data matrix, a scaling matrix element corresponding to the first matrix element in the scaling matrix is determined; a preset operation is performed on the first matrix element and the scaling matrix element, and a second matrix element corresponding to the first matrix element in the second data matrix is determined according to the operation result. The preset operation can be division operation, subtraction operation, mean operation, etc.

[0065] For example, for a first matrix element located at the mth row and the nth column in the first data matrix, an element (i.e., an element at the same position) located at the mth row and the nth column in the scaling matrix is taken as a scaling matrix element corresponding to the first matrix element. Then, the element value of the first matrix element is divided by the element value of the scaling matrix element to obtain the element value of a second matrix element located at the mth row and the nth column in the second data matrix. m and n are natural numbers, and their value ranges are determined according to the matrix size.

[0066] In addition, another embodiment of the present disclosure also provides a neural network-based data processing method. FIG. 2 shows a flowchart of the data processing method. As shown in FIG. 2, the data processing method includes the following steps:

[0067] Step S210: Obtain model data corresponding to a first network layer of a neural network.

[0068] The first network layer of the neural network can be any network layer in the neural network. The model data corresponding to the first network layer can include weight data, parameter data, activation value data, etc. of each neuron in the first network layer, and various types of data that need to be transmitted to other network layers.

[0069] Step S220: Obtain a quantization result of the model data according to a quantization method of the model data.

[0070] The quantization method shown in FIG. 1 can be used to obtain the quantization result of the model data. For details, refer to the related description in the previous embodiment, which will not be repeated here.

[0071] Step S230: Transmit the quantization result of the model data and a scaling matrix to a second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

[0072] The scaling matrix is used to restore the original model data, and its generation method is described in the previous embodiment, which will not be repeated here.

[0073] The second network layer of the neural network can also be any network layer in the neural network. For example, in the forward propagation scenario, the second network layer can be the next network layer of the first network layer; in the backward propagation scenario, the second network layer can be the previous network layer of the first network layer.

[0074] The neural network-based data processing method described above can be applied to both the training process of the neural network and the inference process of the neural network, and the specific scenarios are not limited by the present disclosure.

[0075] In an optional implementation, the second network layer can perform data processing in the following manner: the second network layer performs dequantization processing on the quantization result of the model data to obtain a second data matrix; performs de-scaling processing on the second data matrix according to the scaling matrix to obtain a first data matrix, and performs data processing according to the first data matrix. As can be seen, the processing process of the second network layer is the inverse process of the processing process of the first network layer. The dequantization processing is the inverse process of the quantization processing, and the specific processing manner depends on the selected quantization manner. The de-scaling processing is the inverse process of the scaling processing, and the specific processing manner depends on the specific operation manner of the scaling processing. For example, in the case where the scaling processing adopts the operation manner of element-by-element division, the de-scaling processing can adopt the operation manner of element-by-element multiplication, that is, Hadamard product operation.

[0076] In the above manner, the distribution of the scaled matrix data can be better controlled with the aid of the scaling matrix, thereby improving the quantization effect, significantly reducing the data amount in the transmission process, and improving the model processing efficiency.

[0077] For ease of understanding, the implementation of the model data quantization method provided by the embodiments of the present disclosure will be introduced below by taking an example:

[0078] Large models (LMs), especially large language models (LLMs) such as GPT-3 (Generative Pre-trained Transformer-3) and GLM (General Language Model), have a huge number of parameters exceeding one hundred billion, which brings great challenges to deployment and application. At the same time, with the increasing demand for privacy protection, timeliness, and other needs, the necessity of deploying LLMs locally on terminal devices such as mobile phones and notebook computers is increasing, and therefore, higher requirements are put forward for further reducing the storage space of LLMs and the transmission bandwidth during operation. In the inference scenario of a large language model, the calculation amount of each token is much smaller than the transmission amount of the weight parameters, and therefore, compressing the total storage amount or bandwidth transmission amount of the large language model can greatly improve the inference speed.

[0079] Among them, weight compression is an effective method to solve the above problems. Weight compression mainly involves model quantization, model pruning, low-rank decomposition and other means. For example, model quantization refers to: reducing the amplitude representation precision of model weights, and compressing the 16-bit floating-point number (fp16) representation of weights to 8-bit integer (int8) or 4-bit integer (int4) or lower bit width representation. Compared with the performance of the fp16 model, the int4 precision quantization does not bring obvious decline. For another example, model pruning refers to: discarding unimportant weights in model parameters to achieve sparsification. Low-rank decomposition refers to: taking advantage of the low-rank property of the weight matrix, discarding a part of the singular values and singular vectors that have less influence, and decomposing the large weight matrix into the product of small weight matrices, thereby reducing the parameter storage. Among them, the model pruning and low-rank decomposition methods will significantly reduce the model accuracy, thus greatly affecting the accuracy of the model. Therefore, the model quantization method is usually adopted.

[0080] In related technologies, the model quantization process can be expressed by the following formula:

[0081] Among them, W, are the weight matrices before and after quantization, respectively. Among them, the weight matrix is one of the implementation forms of the first data matrix mentioned above. s is the scaling coefficient, which can generally take the maximum value of the absolute value of W. n is the quantization bit width. Correspondingly, the dequantization process can be expressed by the following formula:

[0082] The above quantization method in related technologies at least has the following defects: Since the scaling coefficient s is generally a number or a vector, that is, it is quantized by row or column, the adjustment effect of the scaling coefficient on the element values in the weight matrix is limited (only the same adjustment processing can be performed on all element values), and the differences between different element values cannot be considered, resulting in unreasonable distribution of the quantized data values (for example, the distribution is not uniform enough), and further resulting in that the average information amount of the weight elements is significantly less than the bit width. In other words, the concentration degree of the quantized data value distribution is not high, the dispersion degree is large, and the information entropy is large, so that the average information amount of the quantized matrix is much smaller than the information amount that can be represented by the bit width. Although the entropy coding and other variable-length techniques can alleviate this problem, the serial decoding nature of entropy decoding greatly limits the decoding speed and limits its application.

[0083] To solve the above problems, the scaling coefficient s in the quantization formula is extended to a scaling matrix in the present example. Since the scaling matrix contains more element values, the control ability of the characteristics (such as distribution) of the quantized data is improved, and the subsequent compression effect is improved while ensuring the accuracy of the quantized values. The key of the present example is that the scaling coefficient s in the quantization formula is extended to a low-rank matrix with the same size as the original weight matrix. At this time, the matrix division operation (for the sake of expression, the present example takes division as an example) can be the element-by-element division of two matrices. Since the scaling coefficient is converted to a scaling matrix, the control ability of each element in the weight matrix is stronger. Since it is a low-rank matrix (such as rank 2), the storage amount of the scaling matrix is much smaller than the data amount of the original weight matrix, thereby achieving the purpose of saving transmission bandwidth.

[0084] Optionally, the quantization formula in the present example can be expressed by the following formula:

[0085] wherein U T V is a low-rank matrix with the same size as W, and the above operations are element-by-element operations (such as element-by-element division) of matrix elements. U T V is used to represent the scaling matrix mentioned above. Since it is division, the scaling matrix U T V can also be called a low-rank normalization matrix. As can be seen, the above formula is used to realize the scaling processing and quantization processing mentioned above. The weight matrix W corresponds to the first data matrix mentioned above. The result of W corresponds to the quantization result mentioned above.

[0086] Correspondingly, the inverse scaling processing and inverse quantization processing mentioned above can be expressed by the following formula:

[0087] wherein the above operations are also element-by-element operations of matrix elements, such as element-by-element multiplication, i.e. Hadamard product. W, and W are the weight matrices before and after quantization respectively. n is the quantization bit width.

[0088] wherein the low-rank normalization matrix (i.e. the scaling matrix) can be determined by at least one of the following two ways:

[0089] (1) Singular value decomposition method: singular value decomposition is performed on the original weight matrix, and the largest part of singular values (for example, 1 or 2) and the corresponding singular vectors are taken to form the scaling matrix.

[0090] (2) Automatic learning method: in the case of a given optimization objective function, for example, the distribution characteristics of the elements after constraint quantization (such as the variance of the second data matrix mentioned above, information entropy, etc.), the parameterized U T V.

[0091] The above two methods can be used alone or in combination. In this example, in order to obtain better results, the above two methods are used in combination: mainly implemented by the automatic learning method, and the initial value required in the automatic learning method is determined by the singular value decomposition method.

[0092] In one specific example, the optimization objective function in the automatic learning method can be set as: minimizing the variance or standard deviation of each matrix element value in the second data matrix obtained after scaling. Wherein, the second data matrix is the matrix obtained after the weight matrix is divided by the scaling matrix. In addition, the variance or standard deviation of each matrix element value in the second data matrix refers to the normalized variance or standard deviation. Wherein, the variance or standard deviation can be calculated after normalizing each matrix element value in the second data matrix according to its maximum absolute value, so as to ensure that the absolute size of the variance or standard deviation can accurately indicate the concentration degree of the data distribution.

[0093] For example, assume that the original weight matrix is 8*8 in size, and the specific data is as follows: W = [[0.1359, -0.1379, -0.3070, -0.1670, -0.5810, -0.2427, -0.1452, 0.1877], [-0.0638, -0.1836, -1.0000, 0.2582, -0.0423, -0.0918, 0.0307, 0.2573], [-0.2738, 0.0902, 0.3298, 0.1328, 0.1847, 0.1690, -0.5393, -0.3133], [0.0664, -0.0187, -0.0928, 0.4233, -0.1629, -0.1390, -0.3258, 0.3423], [-0.5475, 0.4294, -0.2386, -0.6017, -0.1970, 0.3258, -0.0793, -0.1711], [-0.5250, -1.0000, -0.7801, 0.4357, -0.2022, -0.5890, -0.1856, -0.4254], [-0.0518, -0.3237, -0.6431, -0.1681, 0.1042, 0.1742, 0.0892, 0.3444], [0.2199, -0.3837, -0.7674, -0.5518, 0.3983, 0.2780, -0.0985, -0.0996]]

[0094] The standard deviation obtained after normalization of the above weight matrix is 0.3502, which reflects the degree of concentration of the data. The more concentrated the data, the smaller the standard deviation, and thus it is easier to perform further bit width compression processing (i.e., bit width reduction processing) in the subsequent processing process.

[0095] wherein the automatic learning method can be used to determine U T V, assuming that the rank is 2, the size of U T is 8*2, and the size of V is 2*8, and the values thereof are initialized in various ways such as Gaussian distribution. Then, the standard deviation obtained after normalization of the low-rank normalized weight matrix (i.e., the second data matrix) is taken as the objective function, and gradient descent is applied for iterative solution. After 10 iterations, U T= [[0.2500, 0.0938], [1.3140, 1.9981], [-0.3705, 2.2845], [-0.2989, 0.6012], [-0.0168, -0.5043], [0.6487, -2.1925], [-0.4963, -1.3197], [-0.2323, 0.1399]] V = [[-0.8569, 0.3590, -0.5812, -0.0919, 2.0623, 0.0825, 1.4824, -0.4686], [-2.2452, -0.2629, -0.4807, -1.5078, 0.6564, -1.6301, -0.3658, 0.2207]] U T V = [[-0.4248, 0.0651, -0.1904, -0.1644, 0.5772, -0.1323, 0.3363, -0.0965], [-5.6122, -0.0535, -1.7243, -3.1335, 4.0215, -3.1487, 1.2170, -0.1748], [-4.8116, -0.7335, -0.8829, -3.4105, 0.7356, -3.7545, -1.3850, 0.6777], [-1.0937, -0.2654, -0.1153, -0.8791, -0.2218, -1.0048, -0.6631, 0.2727], [1.1465, 0.1265, 0.2522, 0.7619, -0.3657, 0.8206, 0.1596, -0.1034], [4.3667, 0.8092, 0.6770, 3.2463, -0.1015, 3.6276, 1.7638, -0.7877], [3.3882, 0.1687, 0.9229, 2.0354, -1.8899, 2.1102, -0.2530, -0.0586], [-0.1151, -0.1202, 0.0677, -0.1896, -0.3872, -0.2473, -0.3955, 0.1397]]

[0096] Accordingly,

[0097] The "+0.0000001" added in the formula is to prevent division by 0 and improve the stability of numerical calculation. The matrix obtained after the above division operation is the second data matrix, and the standard deviation calculated after normalization of the second data matrix is 0.1773, which is reduced by 50% compared to 0.3502, meaning that after the low-rank normalization of the scaling matrix, the concentration of the data distribution is effectively improved, providing space for subsequent bit width reduction.

[0098] In addition, the present example can also be flexibly adapted to other types of quantization methods. For example, the present example can also be applied to GPTQ (Gradient-based Progressive Tensor Quantization), AWQ (Activation-aware Weight Quantization), and other quantization methods to extend the scaling coefficients in the above quantization methods to scaling matrices, thereby improving the quantization effect of the above quantization methods. In summary, the present disclosure does not limit the specific implementation details of the quantization method. In any scenario that can be quantized by scaling coefficients, the scaling coefficients can be extended to scaling matrices in the scheme of the present example to improve the quantization effect.

[0099] As can be seen, the present example can comprehensively fine-tune the values of the matrix elements through the scaling matrix before quantization, thereby improving uniformity. In addition, the present example is used to process data uniformity in a quantized manner before data storage, and after storage, the original weight data is restored through "dequantization" or "anti-uniformity processing" during the reading and using stage. The specific implementation details of "dequantization" or "anti-uniformity processing" depend on the specific algorithm used during quantization and uniformity processing. For example, in the present example, the above-mentioned formula can be used for "dequantization" or "anti-uniformity processing".

[0100] The core of the present example is to calculate the scaling coefficients of each weight element with as few parameters as possible through a constrained representation (such as low-rank constraint of matrix, etc.), so as to facilitate the compression of quantized data (such as more uniform distribution). This method can greatly improve the control ability of the characteristics of the quantized data with a small amount of additional parameters, while ensuring accuracy, reducing storage and communication requirements after quantization.

[0101] Of course, the present example is not limited to low-rank normalized matrices, but can also flexibly use other compressible matrix representations, such as sparse representation matrices, discrete cosine transform compressed matrices, etc., which are not limited by the present disclosure.

[0102] In addition, the data processing method in the present example can be applied to one or more processing cores in a many-core system. It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without deviating from the principle logic. Due to the limited space, the present disclosure will not be repeated. Those skilled in the art can understand that in the above-mentioned method of the specific embodiment, the specific execution order of each step should be determined according to its function and possible internal logic.

[0103] FIG. 3 is a block diagram of a model data quantization device provided by an embodiment of the present disclosure. Referring to FIG. 3, the present embodiment provides a model data quantization device, which comprises:

[0104] The first acquisition module 31 is adapted to acquire a first data matrix corresponding to the model data;

[0105] The generation module 32 is adapted to generate a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and to perform scaling processing on the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of the matrix element values in the second data matrix;

[0106] The first quantization module 33 is adapted to obtain a quantization result of the model data according to the second data matrix.

[0107] In an optional implementation, the preset constraint condition comprises:

[0108] a first constraint condition for constraining the discrete degree of the matrix element values in the second data matrix; and / or,

[0109] a second constraint condition for constraining the information entropy of the matrix element values in the second data matrix.

[0110] In an optional implementation, the first constraint condition comprises a discrete degree constraint condition for constraining the discrete degree of the matrix element values in the second data matrix; and the second constraint condition comprises an information entropy constraint condition for constraining the information entropy of the matrix element values in the second data matrix; wherein the discrete degree is represented by variance, sum of squares, and / or standard deviation.

[0111] In an optional implementation, the scaling matrix is a low-rank matrix, and the generation module 32 is configured to:

[0112] determine the rank of the scaling matrix according to the matrix size of the first data matrix;

[0113] generate an initial matrix according to the rank of the scaling matrix;

[0114] The initial matrix is iteratively processed according to a preset optimization target, to obtain a scaling matrix corresponding to the first data matrix; wherein the preset optimization target is determined according to the preset constraint condition.

[0115] In an optional implementation, the generation module 32 is configured to:

[0116] The first data matrix is singular value decomposed to extract a preset number of singular values matching a rank of the scaling matrix from the obtained singular values;

[0117] According to the preset number of singular values, a first low-rank matrix and a second low-rank matrix are generated;

[0118] According to a product between a transpose matrix of the first low-rank matrix and the second low-rank matrix, the initial matrix is obtained.

[0119] In an optional implementation, the scaling matrix is a sparse matrix, and the generation module 32 is configured to:

[0120] According to a preset constraint condition, a non-sparse matrix corresponding to the first data matrix is generated;

[0121] According to absolute values of matrix elements in the non-sparse matrix, the non-sparse matrix is sparsified, and a sparse matrix obtained is taken as the scaling matrix corresponding to the first data matrix.

[0122] In an optional implementation, the first quantization module 33 is configured to:

[0123] According to a first bit width, the second data matrix is quantized to obtain an initial quantization result matching the first bit width;

[0124] The initial quantization result is subjected to bit width reduction processing to obtain a target quantization result matching a second bit width; wherein the second bit width is smaller than the first bit width.

[0125] In an optional implementation, the model data includes: weight data and / or activation value data in a neural network.

[0126] Wherein, the matrix size of the scaling matrix is the same as the matrix size of the first data matrix, and the generation module 32 is configured to:

[0127] For any first matrix element in the first data matrix, a scaling matrix element corresponding to the first matrix element in the scaling matrix is determined;

[0128] performing a preset operation on the first matrix element and the scaling matrix element, and determining an element value of a second matrix element corresponding to the first matrix element in the second data matrix according to an operation result.

[0129] FIG. 4 is a block diagram of a data processing apparatus based on a neural network according to an embodiment of the present disclosure. Referring to FIG. 4, the data processing apparatus according to an embodiment of the present disclosure includes:

[0130] The second obtaining module 41 is adapted to obtain model data corresponding to a first network layer of the neural network.

[0131] The second quantization module 42 is adapted to obtain a quantization result of the model data according to the quantization method in any one of claims 1-7.

[0132] The transmission module 43 is adapted to transmit the quantization result of the model data and a scaling matrix to a second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

[0133] In an optional implementation, the data processing performed by the second network layer according to the quantization result of the model data and the scaling matrix includes: the second network layer performs dequantization processing on the quantization result of the model data to obtain a second data matrix; and the second network layer performs de-scaling processing on the second data matrix according to the scaling matrix to obtain the first data matrix, and performs data processing according to the first data matrix.

[0134] FIG. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0135] Referring to FIG. 5, the electronic device according to an embodiment of the present disclosure includes at least one processor 901, at least one memory 902, and one or more I / O interfaces 903 connected between the processor 901 and the memory 902. The memory 902 stores one or more computer programs executable by the at least one processor 901. The one or more computer programs are executed by the at least one processor 901, so that the at least one processor 901 can perform the model training method described above.

[0136] In some embodiments, the processing device can be a brain-like chip. Since the brain-like chip can use vectorized computing, and needs to load weight information and other parameters of the neural network model from an external memory such as a double data rate (DDR) synchronous dynamic random access memory, the batch processing operation efficiency is higher in the embodiment of the present disclosure.

[0137] The embodiments of the present disclosure further provide a computer readable storage medium, which stores a computer program, wherein the computer program implements the model training method described above when executed by a processor / processing core. The computer readable storage medium can be a volatile or non-volatile computer readable storage medium.

[0138] The embodiments of the present disclosure further provide a computer program product comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, which, when run in a processor of an electronic device, causes the processor in the electronic device to perform the method described above.

[0139] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, the functions of the modules / units in the systems and devices can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementation, the split between the functional modules / units referred to in the above description does not necessarily correspond to the split between physical components; for example, one physical component can serve as multiple functional modules or a single function or step can be performed by several physical components working together. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer readable storage media, which can include computer storage media (or non-transitory media) and communication media (or transitory media).

[0140] As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable program instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Furthermore, it is well known to those of ordinary skill in the art that communication media typically embodies computer readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. In this document, the term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media.

[0141] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0142] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0143] The computer program product described herein can be embodied specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied specifically as a computer storage medium, and in another optional embodiment, the computer program product is embodied specifically as a software product, such as a software development kit (SDK) or the like.

[0144] The computer program product described herein can be embodied specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied specifically as a computer storage medium, and in another optional embodiment, the computer program product is embodied specifically as a software product, such as a software development kit (SDK) or the like.

[0145] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions which execute via the one or more processors of the computer or other programmable data processing devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0146] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0147] The flow and block diagrams in the drawings show architectural, functional, and operational aspects of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of instructions which comprise one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may

[0148] Example embodiments have been disclosed and, although a specific terminology is employed, it is merely for the convenience of the reader and is not intended to be limiting. In some instances, specific details have been included for the purpose of providing a thorough understanding of the embodiments presented, but it will be apparent to those skilled in the art that embodiments of the application can be practiced without these specific details. In some instances, features, characteristics and / or elements described in connection with a particular embodiment can be used in conjunction with, or switched to, features, characteristics and / or elements described in connection with other embodiments, unless otherwise explicitly stated. Accordingly, it will be understood that various changes in form and details can be made without departing from the scope of the disclosure as set forth in the appended claims.

Claims

1. A method for quantizing model data, the method comprising: obtaining a first data matrix corresponding to the model data; generating a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and performing scaling processing on the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain a distribution feature of matrix element values in the second data matrix; and obtaining a quantization result of the model data according to the second data matrix. The preset constraint condition comprises: a first constraint condition used to constrain a discrete degree of the matrix element values in the second data matrix; and / or a second constraint condition used to constrain an information entropy of the matrix element values in the second data matrix. The first constraint condition comprises a discrete degree constraint condition used to constrain the discrete degree of the matrix element values in the second data matrix; and the second constraint condition comprises an information entropy constraint condition used to constrain the information entropy of the matrix element values in the second data matrix; wherein the discrete degree is represented by variance, sum of squares, and / or standard deviation. The scaling matrix is a low-rank matrix, and the generating of the scaling matrix corresponding to the first data matrix according to the preset constraint condition comprises: determining a rank of the scaling matrix according to a matrix size of the first data matrix; generating an initial matrix according to the rank of the scaling matrix; and performing iterative processing on the initial matrix according to a preset optimization target to obtain the scaling matrix corresponding to the first data matrix; wherein the preset optimization target is determined according to the preset constraint condition.

2. The method of claim 1, wherein, The generating of the initial matrix according to the rank of the scaling matrix comprises: performing singular value decomposition processing on the first data matrix, extracting a preset number of singular values matched with the rank of the scaling matrix from a plurality of singular values obtained; generating a first low-rank matrix and a second low-rank matrix according to the preset number of singular values; and obtaining the initial matrix according to a product between a transpose matrix of the first low-rank matrix and the second low-rank matrix. The scaling matrix is a sparse matrix, and the generating of the scaling matrix corresponding to the first data matrix according to the preset constraint condition comprises: generating a non-sparse matrix corresponding to the first data matrix according to the preset constraint condition; and performing sparsification processing on the non-sparse matrix according to absolute values of matrix elements in the non-sparse matrix, and taking a sparse matrix obtained as the scaling matrix corresponding to the first data matrix. The obtaining of the quantization result of the model data according to the second data matrix comprises: performing quantization processing on the second data matrix according to a first bit width to obtain an initial quantization result matched with the first bit width; and performing bit width reduction processing on the initial quantization result to obtain a target quantization result matched with a second bit width; wherein the second bit width is smaller than the first bit width.

3. The method of claim 2, wherein, The model data comprises weight data and / or activation value data in a neural network.

4. The method according to any of claims 1 to 3, wherein, The matrix size of the scaling matrix is the same as the matrix size of the first data matrix, and the performing of the scaling processing on the first data matrix according to the scaling matrix to obtain the second data matrix comprises: ​ ​ ​ 5. The method of claim 4, wherein, ​ ​ ​ ​ 6. The method of any one of claims 1-3, wherein, ​ ​ ​ 7. The method of any one of claims 1-3, wherein, ​ ​ ​ 8. The method of any one of claims 1-3, wherein, ​ ​ For any first matrix element in the first data matrix, a scaling matrix element corresponding to the first matrix element in the scaling matrix is determined; For the first matrix element and the scaling matrix element, a preset operation is performed, and an element value of a second matrix element corresponding to the first matrix element in the second data matrix is determined according to an operation result.

9. The method of claim 8, wherein, If the preset operation includes a division operation, the quantization result of the model data is obtained by the following formula: wherein W for characterizing the first data matrix, U T V for characterizing the scaling matrix, n for characterizing the quantization bit width at the time of quantization.

10. A neural network-based data processing method, comprising: obtaining model data corresponding to a first network layer of the neural network; obtaining a quantization result of the model data according to the quantization method in any one of claims 1-9; transmitting the quantization result of the model data and the scaling matrix to a second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

11. The method of claim 10, wherein, The data processing performed by the second network layer according to the quantization result of the model data and the scaling matrix comprises: The second network layer performs dequantization processing on the quantization result of the model data to obtain a second data matrix; performing de-scaling processing on the second data matrix according to the scaling matrix to obtain the first data matrix, and performing data processing according to the first data matrix.

12. The method of claim 11, wherein, The first data matrix is obtained by the following formula: wherein quantization results for characterizing the model data, W for characterizing the first data matrix, U T V for characterizing the scaling matrix, n for characterizing the dequantization bit width at dequantization.

13. A model data quantization apparatus, comprising: a first obtaining module adapted to obtain a first data matrix corresponding to the model data; a generating module adapted to generate a scaling matrix corresponding to the first data matrix according to a preset constraint condition, and perform scaling processing on the first data matrix according to the scaling matrix to obtain a second data matrix; wherein the preset constraint condition is used to constrain the distribution characteristics of matrix element values in the second data matrix; a first quantization module adapted to obtain a quantization result of the model data according to the second data matrix.

14. A neural network-based data processing apparatus, comprising: a second obtaining module adapted to obtain model data corresponding to a first network layer of the neural network; a second quantization module adapted to obtain a quantization result of the model data according to the quantization method in any one of claims 1-9; a transmission module adapted to transmit the quantization result of the model data and the scaling matrix to a second network layer of the neural network, so that the second network layer performs data processing according to the quantization result of the model data and the scaling matrix.

15. An electronic device, comprising: comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-12.

17. A computer program product comprising computer readable code, or a non-transitory computer readable storage medium having computer readable code embodied thereon, wherein, When the computer readable code is running in the processor of the electronic device, the processor in the electronic device performs the method of any one of claims 1-12.

Images