Data processing method and device, readable medium, electronic equipment and program product

Abstract

The invention relates to a data processing method and device, a readable medium, electronic equipment and a program product, and can carry out elastic combination quantification on a weight matrix in a model network layer so as to consider the precision of a model and memory consumption. The method comprises the steps that a target weight matrix of a network layer in a neural network model is determined, the importance degree of each element in the target weight matrix is determined, and the importance degree of each element is in positive correlation with the influence degree of the element on the model precision of the neural network model; determining significant elements and non-significant elements in the target weight matrix according to the importance degree, wherein the importance degree of the significant elements is higher than that of the non-significant elements; the significant elements are quantized to the first bit width, the non-significant elements are quantized to the second bit width, a weight quantization matrix is obtained, the first bit width is larger than the second bit width, and the weight quantization matrix is used for processing input data of the network layer to obtain output data of the network layer.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data processing method, apparatus, readable medium, electronic device, and program product. Background Technology

[0002] The accuracy of a neural network model is directly proportional to the quantization bit width of the weight matrix. The larger the quantization bit width of the weight matrix, the higher the model accuracy, but the more memory the model occupies.

[0003] In related technologies, quantizing the weight matrix can compress the model and save memory and computational resources. For example, binarizing the weight matrix, i.e., quantizing one weight value to one bit, can significantly reduce the model's accuracy, making it impossible to balance model accuracy and memory consumption. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the subsequent detailed description section. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, this disclosure provides a data processing method, the data processing method comprising: Determine the target weight matrix of the network layer in the neural network model, and determine the importance of each element in the target weight matrix. The importance of the element is positively correlated with the degree of influence of the element on the model accuracy of the neural network model. Based on the degree of importance, significant and insignificant elements in the target weight matrix are determined, wherein the significance of the significant elements is higher than that of the insignificant elements; The significant elements are quantized to the first bit width, and the non-significant elements are quantized to the second bit width to obtain a weight quantization matrix. The first bit width is greater than the second bit width. The weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

[0006] Secondly, this disclosure provides a data processing apparatus, the data processing apparatus comprising: The first determining module is used to determine the target weight matrix of the network layer in the neural network model, and to determine the importance of each element in the target weight matrix. The importance of the element is positively correlated with the degree of influence of the element on the model accuracy of the neural network model. The second determining module is used to determine significant and insignificant elements in the target weight matrix according to the importance level, wherein the importance level of the significant elements is higher than that of the insignificant elements; The quantization module is used to quantize the significant elements to the first bit width and the non-significant elements to the second bit width to obtain a weight quantization matrix. The first bit width is greater than the second bit width. The weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

[0007] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in any of the first aspects above.

[0008] Fourthly, this disclosure provides an electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method described in any of the first aspects above.

[0009] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0010] The above technical solution divides the elements of the target weight matrix of the network layer in the neural network model into salient elements with high importance and insignificant elements with low importance. By quantizing the salient elements to a high bit width and the insignificant elements to a low bit width, a combined quantized weight matrix is ​​obtained. That is, different elements in the weight matrix of the same network layer can be quantized to different bit widths. By quantizing the salient elements that have a high impact on model accuracy to a high bit width and the insignificant elements that have a low impact on model accuracy to a low bit width, flexible combined quantization of the weight matrix is ​​achieved, so as to balance model accuracy and memory consumption.

[0011] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure; Figure 2a This is a schematic diagram illustrating a target weight matrix according to an exemplary embodiment of the present disclosure; Figure 2b This is a schematic diagram illustrating a weight quantization matrix according to an exemplary embodiment of the present disclosure; Figure 2c This is a schematic diagram illustrating a fine-tuned weight quantization matrix according to an exemplary embodiment of the present disclosure; Figure 3 This is a structural block diagram of a data processing apparatus according to an exemplary embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0015] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] A Large Language Model (LLM) is a language model based on deep learning techniques, trained through unsupervised learning on a large amount of text. LLMs typically have a very large number of parameters and can handle a wide variety of natural language tasks, such as text generation, question answering, reasoning computation, and reading comprehension.

[0025] The sheer number of parameters presents significant deployment challenges for memory-constrained devices. To reduce model memory consumption, the weight matrix is ​​typically quantized. Quantization not only compresses the model and saves memory resources, but also improves memory access efficiency and computation speed. Related techniques can quantize model weights to 8 bits with near-lossless accuracy; however, quantization below 4 bits significantly reduces model accuracy, thus making it impossible to balance model accuracy and memory consumption.

[0026] In view of the above, this disclosure provides a data processing method, apparatus, readable medium, electronic device, and program product to solve the aforementioned technical problems.

[0027] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 1 The data processing method includes: S101: Determine the target weight matrix of the network layers in the neural network model, and determine the importance of each element in the target weight matrix.

[0029] The importance of an element is positively correlated with its impact on the accuracy of the neural network model.

[0030] It is worth noting that the neural network model can be a deep learning model trained on massive amounts of text data, such as text generation models, text recognition models, and the aforementioned large language models; it can also be a deep learning model trained on massive amounts of image data, such as image generation models and image recognition models. The specific model can be determined according to requirements, and this disclosure does not impose any restrictions. The neural network model can include multiple network layers, and each network layer can include a weight matrix for processing the input data of that network layer, such as feature extraction and convolution processing.

[0031] For example, if elements in the weight matrix whose values ​​significantly exceed the average weight are quantized with a low bit width, it will significantly affect the model accuracy of the neural network model. That is, the greater the difference between the weight of an element and the average weight, the greater the impact on the model accuracy of the neural network model, and correspondingly, the greater the importance of the element.

[0032] S102: Determine the significant and insignificant elements in the target weight matrix based on their importance, with significant elements having a higher importance than insignificant elements.

[0033] For example, if the significance of significant elements is higher than that of insignificant elements, it means that significant elements have a greater impact on model accuracy than insignificant elements.

[0034] S103: Quantize significant elements to the first bit width and non-significant elements to the second bit width to obtain the weight quantization matrix.

[0035] In this model, the first bit width is greater than the second bit width. The weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

[0036] In the possible configuration, the first bit is 8 bits wide and the second bit is 1 bit wide.

[0037] For example, taking an M-layer convolutional neural network as an example, a triplet can be used. <W m I m-1 , *> represents the m-th convolutional layer, where C represents the weight of the m-th layer. in C represents the number of input channels in the m-th layer. out This represents the number of output channels in the m-th layer, and K represents the number of convolutional kernels in the m-th layer. This can be achieved through... This represents the k-th convolutional kernel in the m-th layer. This represents the input data of the m-th layer (i.e., the output data of the previous layer), w and h represent the width and height of the input data, respectively, and * represents the convolution operation based on multiplication and accumulation.

[0038] If the above weights are directly binarized (quantized to 1 bit), the binarized weights are obtained. In the case of neural network models with a very large number of parameters, such as large language models, the model performance will be worse than random guessing. This is because there are large errors in solving the scaling coefficients or truncation of the significant elements in the weight matrix during the quantization process. In other words, directly binarizing the significant elements will lead to a significant decrease in model accuracy.

[0039] As can be seen from the above, the model accuracy can be guaranteed even when the model weights are quantized to 8 bits. Therefore, significant elements that have a high impact on model accuracy are quantized to 8 bits, while non-significant elements that have a low impact on model accuracy are binarized. This allows the model to be effectively compressed without affecting its accuracy, thereby saving memory resources.

[0040] Furthermore, the first bit width and the second bit width can be determined according to specific memory requirements or precision requirements, such as the first bit width being 16 bits and the second bit width being 4 bits, etc. This disclosure does not impose any restrictions on this.

[0041] Using the above method, the elements of the target weight matrix of the network layer in the neural network model are divided into salient elements with high importance and insignificant elements with low importance. By quantizing the salient elements to a high bit width and the insignificant elements to a low bit width, a combined quantized weight matrix is ​​obtained. That is, different elements in the weight matrix of the same network layer can be quantized to different bit widths. By quantizing the salient elements that have a high impact on model accuracy to a high bit width and the insignificant elements that have a low impact on model accuracy to a low bit width, flexible combined quantization of the weight matrix is ​​achieved, so as to balance model accuracy and memory consumption.

[0042] The following example illustrates the process of determining the importance of elements in the weight matrix.

[0043] One possible approach to determining the importance of each element in the target weight matrix may include: setting each element in the target weight matrix to 0 to obtain a weight adjustment matrix, and determining the importance of the elements based on the target weight matrix and the weight adjustment matrix.

[0044] For example, to determine the importance of a certain element in the weight matrix, the element can be set to 0, and then the impact of the target weight matrix and the weight adjustment matrix before and after setting the element to 0 on the model's loss function can be compared.

[0045] For example, to simplify the expression, let's take the weight matrix of the x-th layer in the neural network model as an example. Taking an example where the weight matrix of layer X is an M-row N-column matrix, an adjustment matrix can be applied to the weight matrix W. The weight adjustment matrix is ​​obtained by adjusting the weight matrix W. It can be expressed in vector form, with W q This represents the element at position q in the weight vector. Taking the adjustment of the element at position q in the vector as an example, W q + =0, that is, apply a parameter to each element to make the element set to 0, and obtain the weight adjustment matrix corresponding to the element, where q∈[0,...,M×N-1].

[0046] In one possible approach, determining the importance of an element based on the target weight matrix and the weight adjustment matrix may include: determining a first loss value for the network layer in the neural network model using the target weight matrix and a second loss value using the weight adjustment matrix, respectively; determining the loss difference between the second loss value and the first loss value, and using the loss difference as the importance of the element.

[0047] For example, the loss difference between the second loss value and the first loss value can be determined based on the following formula. :

[0048] Where L represents the loss function of the neural network model, g represents the gradient of the parameters in the loss function, corresponding to first-order information, H represents the Hessian matrix of the target weight matrix, corresponding to second-order information, and O represents the omission of generalization symbols.

[0049] For example, based on the above calculation formula, the difference in loss between the target weight matrix and the weight adjustment matrix before and after each element is set to 0 can be obtained. This difference in loss is used as the importance of the element, so that the element can be determined as a significant element or a non-displayed element according to its importance in the future.

[0050] In some possible approaches, the data processing method further includes: determining the first Hessian matrix of the target weight matrix, and determining the first inverse matrix of the Hessian matrix. Determining the importance of each element in the target weight matrix includes: for each element in the target weight matrix, determining the first element value of the element, and determining the second element value of the target element in the first inverse matrix; the square of the first element value divided by the second element value is taken as the importance of the element; the target element is the element in the i-th row and i-th column of the first inverse matrix, where i is obtained by multiplying the row number and column number of the element in the target weight matrix and then subtracting 1.

[0051] It is worth noting that for a model that has converged after training, the gradient is close to 0 when the parameter vector is in a local minimum. Therefore, the gradient term can be ignored, as can the cubic term, resulting in the following calculation formula:

[0052] For example, with W i This represents the element at position i in the weight vector. Taking the adjustment of the element at position i in the vector as an example, in order to set the element at position i in the weight vector W to 0, Minimum can be expressed as constraint W i + =0. For constrained optimization problems, we can construct a Lagrange function by introducing Lagrange multipliers λ, transforming it into an unconstrained problem. Constructing the function L, we obtain the following computational formula:

[0053] Furthermore, to facilitate calculation, e can be introduced. i Let i represent a unit vector where position i is 1 and all other positions are 0. Then the above calculation formula can be transformed into:

[0054] For example, in order to solve a suitable The problem of minimizing function L (by setting the element at position i to 0) can be transformed into finding the minimum point of function L. Therefore, by taking the partial derivative of function L with respect to ΔW and setting it to 0 (solving for the extreme point), we can obtain the following calculation formula 2:

[0055] Because of e q Let H represent a unit vector where the value at position i is 1 and the values ​​at all other positions are 0. The inverse of the Hessian matrix H is a square matrix. The element at position i is Therefore, we can obtain Combined with constraint W i + =0, so we can obtain the following calculation formula 3:

[0056] Substituting the above calculation formula 3 into calculation formula 2, we obtain the following calculation formula 4:

[0057] in, Let represent the element in the i-th column of the inverse of the Hessian matrix H. Equation 4 can be understood as the adjustment required to the weight matrix W, assuming the element at position i is set to 0 and its impact on the loss function of the neural network model is minimized.

[0058] Furthermore, substituting formula 2 into formula 1 yields the result. Updating the weight matrix results in an increase in the loss of the neural network model.

[0059] Further simplification yields:

[0060] in, It is the identity matrix. Substituting formula 3 into the simplified formula above, we obtain formula 5 as follows:

[0061] The above calculation formula 5 can be understood as the actual impact of setting the element at position i of the weight matrix W to 0 on the loss function of the neural network model. The larger this value is, the higher the importance of the element at position q of the weight matrix W.

[0062] In summary, we can select Let i ∈ [0, ..., M×N-1] represent the importance of each element in the weight matrix W. Therefore, the importance of an element can be derived from its value in the target weight matrix, eliminating the need to solve for the model's loss error before and after setting elements to 0, simplifying the calculation process and improving model quantization efficiency.

[0063] The above embodiments can determine the importance of each element in the weight matrix. The following embodiments illustrate the process of dividing the weight matrix into significant and insignificant elements.

[0064] As Figure 2a As can be seen from the importance of each element in the weight matrix shown, the significant weights (elements marked with black boxes) in the weight matrix are distributed randomly and uniformly. Since there is no discernible column-wise pattern in the distribution of significant elements, using a column-wise method to select significant elements may impair model performance. Therefore, an element-wise method can be used to filter significant elements.

[0065] In one possible approach, determining significant and insignificant elements in the target weight matrix based on importance may include: determining a first number of importance levels and a second number of importance levels from the importance levels, where each of the second number of importance levels is higher than the first number of importance levels, the first number being equal to the product of the total number of elements in the target weight matrix and a first preset ratio, the second number being equal to the product of the total number of elements and a second preset ratio, and the sum of the first preset ratio and the second preset ratio being equal to 1; determining the elements corresponding to the first number of importance levels as insignificant elements, and determining the elements corresponding to the second number of importance levels as significant elements.

[0066] For example, the proportion of non-significant elements can be set to β, then the proportion of significant elements is 1-β. By adjusting the value of β, the bit width of the quantized weight matrix can be flexibly adjusted, thereby adjusting the memory resources occupied by the model. The value of β can be set according to requirements, and this disclosure does not impose any restrictions on it.

[0067] The weight matrix of the x-th layer in the neural network model For example, determine the prominent elements according to the set proportions. Insignificant elements ,in .

[0068] Furthermore, taking an example where the quantization width of significant elements is 8 bits, the quantization width of insignificant elements is 1 bit, and the proportion of insignificant elements is β, then the overall equivalent quantization width N is... bit It can be expressed as follows: N bit≤1×β+8×(1-β)+1 Where 1×β represents the equivalent bit width of non-significant elements, 8×(1-β) represents the equivalent bit width of significant elements, and the third term "1" is the maximum bit width required to store the position information of each significant element. Using CSR (compressed sparse row format) and CSC (compressed sparse column format) storage methods, the storage bit width for each significant element does not require 1 bit. N bit Less than or equal to the value on the right side of the expression.

[0069] As can be seen from the above expression, when β=0, the overall equivalent quantization bit width is 9 bits, that is, all elements of the weight matrix are quantized to 8 bits. When β=0, the overall equivalent quantization bit width is 2 bits, that is, all elements of the weight matrix are binarized. Therefore, by adjusting the value of β, the overall equivalent quantization bit width can be flexibly adjusted, and thus the memory resources occupied by the model can be adjusted.

[0070] The following example illustrates the process of quantizing the weight matrix.

[0071] It is worth noting that the GPTQ algorithm (Post-Training Quantization for GPTModels) can be used to quantize salient elements to the first bit width and insignificant elements to the second bit width, using the weight matrix W. For example, in the quantification process, it can be done by column (C in (Dimension) Iteration.

[0072] In one possible approach, quantizing significant elements to the first bit width and non-significant elements to the second bit width to obtain a weight quantization matrix can include: using the elements corresponding to the first column of the target weight matrix as target column elements; quantizing significant elements in the target column elements to the first bit width and non-significant elements in the target column elements to the second bit width to obtain quantized column elements; determining a residual compensation matrix based on the target weight matrix, target column elements, and quantized column elements, the residual compensation matrix being used to compensate for the quantization error between the target column elements and the quantized column elements to the unquantized residual weight matrix in the target weight matrix; adding the residual compensation matrix and the residual weight matrix to obtain a new target weight matrix, and using the elements corresponding to the first column of the new target weight matrix as new target column elements, repeating the steps of quantizing significant elements in the target column elements to the first bit width and non-significant elements in the target column elements to the second bit width to obtain quantized column elements, until the last column of the target weight matrix is ​​quantized, thus obtaining the weight quantization matrix.

[0073] For example, the columns in the weight matrix W are iteratively quantized. Starting from the first column of the weight matrix W, insignificant elements are binarized, and significant elements are quantized using 8 bits. Based on the quantized first column, the remaining unquantized weight matrix W1 is determined as the residual compensation matrix. 1. Residual Compensation Matrix 1. The remaining unquantized weight matrix W1 is of the same type. The remaining compensation matrix... 1 is applied to the remaining unquantized weight matrix W1, i.e. The new target weight matrix is ​​obtained by adding 1+W1. Then, the insignificant elements in the first column of the new target weight matrix are binarized, and the significant elements are quantized using 8 bits. This process is repeated until the entire weight matrix is ​​quantized. For example, from... Figure 2a The weight matrix shown is quantized to Figure 2b The weight quantization matrix shown is used to compensate for the quantization error of each column in the remaining unquantized weight matrix using a compensation matrix, thereby reducing the overall quantization error of the weight matrix.

[0074] In one possible approach, determining the residual compensation matrix based on the target weight matrix, target column elements, and quantized column elements may include: determining the second Hessian matrix of the target weight matrix and determining the second inverse matrix of the Hessian matrix; for each element in the target column elements, determining the difference between the element and its corresponding quantized element in the quantized column elements, dividing the difference by the value of the third element of the element in the j-th row and j-th column of the second inverse matrix, and then multiplying it by all elements in the j-th column of the second inverse matrix to obtain the element compensation matrix, where j is obtained by multiplying the row number and column number of the element in the target weight matrix and then subtracting 1; and summing the element compensation matrices corresponding to all elements in the target column elements to obtain the residual compensation matrix.

[0075] For example, taking an M×N weight matrix W as an example, the element w at position j in the weight vector corresponding to the weight matrix W... j Quantified as For j∈[0,...,M×N-1], by minimizing the single-layer reconstruction error, the element compensation matrix applied to the remaining weight matrix can be obtained. -j :

[0076] Where -j represents the remaining unquantized weight elements excluding position j, This represents the element in the j-th row and j-th column of the inverse of the Hessian matrix corresponding to the target weight matrix. This represents all elements in the j-th column of the inverse matrix. It should be noted that during column-wise quantization, after each quantization to obtain a new target weight matrix, the inverse of the Hessian matrix corresponding to the new target weight matrix needs to be recalculated.

[0077] It should be noted that element w j The single-layer error caused by quantization is .

[0078] Furthermore, for each element in each quantized column, an element compensation matrix is ​​calculated. Then, the element compensation matrices corresponding to all elements in that column are summed to obtain the residual compensation matrix of the remaining unquantized weight matrix for that column. This allows the quantization error of each column to be compensated for in the remaining unquantized weight matrix, thereby reducing the overall quantization error of the weight matrix.

[0079] It's worth noting that the quantization process described above can be directly applied to PTQ (Post-Training Quantization). If it needs to be applied to the QAT (Quant Aware Training) framework, quantization-aware training can be used to further improve model performance. To improve model training efficiency, salient elements can be fixed during training, while the scaling factors of non-salient binarized elements can be fine-tuned, thus balancing high efficiency and high accuracy of the model.

[0080] In one possible approach, significant elements are quantized to the first bit width and insignificant elements are quantized to the second bit width to obtain a weighted quantization matrix, including: quantizing significant elements to the first bit width and insignificant elements to the second bit width to obtain an initial quantization matrix; multiplying each insignificant element in the initial quantization matrix by a preset scaling factor of the column containing the insignificant element to obtain a weighted quantization matrix.

[0081] For example, each insignificant element in the initial quantization matrix is ​​multiplied by a preset scaling factor of the column containing the insignificant element, such as... Figure 2b The initial quantization matrix shown is fine-tuned as follows: Figure 2c The weight quantization matrix shown.

[0082] In one possible approach, the preset scaling factor is determined as follows: for each column in the target weight matrix, the scaling factor variable that minimizes the L2 norm between the target weight matrix and the target product matrix is ​​determined as the target scaling factor for that column; wherein, the target product matrix is ​​obtained by multiplying each insignificant element in the initial quantization matrix by the scaling factor variable of the column containing the insignificant element.

[0083] For example, continuing with the M-layer convolutional neural network mentioned above, assume that the binarized weights corresponding to the non-significant elements... The corresponding floating-point scaling factor , Representing floating-point values, floating-point weights can be... It can be approximated by binarized weights and floating-point scaling factors:

[0084] Transform into an optimization problem:

[0085] Among them, || || 2 If we calculate the L2 norm, then the solution to the optimization problem is:

[0086] Among them, || || l1 This indicates the calculation of the L1 norm value. , indicating floating-point weight The number of elements is such that, since the Sign function is not differentiable, the STE function can be used for differentiable approximation during training.

[0087] Furthermore, the weight quantization matrix after fine-tuning at the m-th layer can be expressed as:

[0088] in, =[ ], =[ ].

[0089] This allows us to determine the scaling factor corresponding to each column in the weight matrix, and then fine-tune the initial quantization matrix to obtain the final weight quantization matrix, thereby further reducing quantization error and improving the model accuracy of the neural network model.

[0090] During QAT training, the significant elements in the weight matrix can be fixed, and the floating-point scaling factor 'a' of the non-significant elements can be used as trainable parameters. The model can be initialized based on the solution of the above optimization problem, and a small amount of training can be performed to obtain a trained neural network model.

[0091] By employing the above method, significant and insignificant elements in the weight matrix of the neural network layers are distinguished. Significant elements are quantized with a high bit width to enhance language capabilities, while insignificant elements are quantized with a low bit width, such as binarization. This achieves flexible combinatorial quantization (adjustable overall quantization bit width) while maintaining high model accuracy. Furthermore, during QAT training, a strategy of fixing significant elements and fine-tuning the binarization floating-point scaling factor is used to further improve model accuracy.

[0092] It should be understood that the above calculation formula is only illustrative and can be modified in practical applications. (Additionally attached...) Figures 2a-2c The specific values ​​in the figures are for illustrative purposes only and are not intended to be limiting.

[0093] Based on the same concept, embodiments of this disclosure also provide a data processing apparatus, such as... Figure 3 As shown, the data processing device 300 may include: The first determining module 301 is used to determine the target weight matrix of the network layer in the neural network model, and to determine the importance of each element in the target weight matrix, wherein the importance of the element is positively correlated with the degree of influence of the element on the model accuracy of the neural network model; The second determining module 302 is used to determine significant and insignificant elements in the target weight matrix according to the importance level, wherein the importance level of the significant elements is higher than that of the insignificant elements; The quantization module 303 is used to quantize the significant elements to the first bit width and the non-significant elements to the second bit width to obtain a weight quantization matrix. The first bit width is greater than the second bit width. The weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

[0094] Using the aforementioned apparatus, the elements of the target weight matrix of the network layer in the neural network model are divided into salient elements with high importance and insignificant elements with low importance. By quantizing the salient elements to a high bit width and the insignificant elements to a low bit width, a combined quantized weight matrix is ​​obtained. That is, different elements in the weight matrix of the same network layer can be quantized to different bit widths. By quantizing the salient elements that have a high impact on model accuracy to a high bit width and the insignificant elements that have a low impact on model accuracy to a low bit width, flexible combined quantization of the weight matrix is ​​achieved, so as to balance model accuracy and memory consumption.

[0095] Optionally, the first determining module 301 is used to: For each element in the target weight matrix, the element is set to 0 to obtain a weight adjustment matrix. The importance of the element is determined based on the target weight matrix and the weight adjustment matrix.

[0096] Optionally, the first determining module 301 is used to: Determine the first loss value of the network layer in the neural network model using the target weight matrix and the second loss value using the weight adjustment matrix, respectively. Determine the loss difference between the second loss value and the first loss value, and use the loss difference as the importance of the element.

[0097] Optionally, the data processing device 300 further includes: The third determining module is used to determine the first Hessian matrix of the target weight matrix and the first inverse matrix of the Hessian matrix; The first determining module 301 is used for: For each element in the target weight matrix, determine the first element value of the element and determine the second element value of the target element in the first inverse matrix. The square of the first element value is divided by the second element value to obtain the division result as the importance of the element. The target element is the element in the i-th row and i-th column of the first inverse matrix, where i is obtained by multiplying the row number and column number of the element in the target weight matrix and then subtracting 1.

[0098] Optionally, the quantization module 303 includes: The target determination module is used to take the element corresponding to the first column of the target weight matrix as the target column element; The column quantization module is used to quantize the significant elements in the target column elements to the first bit width and to quantize the non-significant elements in the target column elements to the second bit width, so as to obtain quantized column elements; The compensation matrix determination module is used to determine the remaining compensation matrix based on the target weight matrix, the target column elements and the quantization column elements. The remaining compensation matrix is ​​used to compensate the quantization error between the target column elements and the quantization column elements to the unquantized remaining weight matrix in the target weight matrix. The repeated execution module is used to add the remaining compensation matrix and the remaining weight matrix to obtain a new target weight matrix, and take the element corresponding to the first column of the new target weight matrix as the new target column element, and repeatedly execute the steps of quantizing the significant elements in the target column element to the first bit width and quantizing the non-significant elements in the target column element to the second bit width to obtain quantized column elements, until the last column element in the target weight matrix is ​​quantized to obtain the weight quantization matrix.

[0099] Optionally, the compensation matrix determination module includes: The fourth determining module is used to determine the second Hessian matrix of the target weight matrix and to determine the second inverse matrix of the Hessian matrix; The fifth determining module is used to determine, for each element in the target column elements, the difference between the element and the corresponding quantized element in the quantized column elements, divide the difference by the third element value of the element in the j-th row and j-th column of the second inverse matrix, and then multiply it by all elements in the j-th column of the second inverse matrix to obtain the element compensation matrix, wherein j is obtained by multiplying the row number and column number of the element in the target weight matrix and then subtracting 1; The accumulation module is used to accumulate the element compensation matrices corresponding to all elements in the target column to obtain the remaining compensation matrix.

[0100] Optionally, the quantization module 303 includes: An initial quantization module is used to quantize the significant elements to the first bit width and the non-significant elements to the second bit width to obtain an initial quantization matrix; The scaling quantization module is used to multiply each insignificant element in the initial quantization matrix by a preset scaling factor of the column containing the insignificant element to obtain the weighted quantization matrix.

[0101] Optionally, the preset scaling factor is determined in the following way: For each column of the target weight matrix, the scaling factor variable that minimizes the L2 norm between the target weight matrix and the target product matrix is ​​determined as the target scaling factor for that column. The target product matrix is ​​obtained by multiplying each insignificant element in the initial quantization matrix by the scaling factor variable of the column containing the insignificant element.

[0102] Optionally, the second determining module 302 is used to: From the importance level, a first number of importance levels and a second number of importance levels are determined, each of the second number of importance levels being higher than the first number of importance levels. The first number is equal to the product of the total number of elements in the target weight matrix and a first preset ratio, and the second number is equal to the product of the total number of elements and a second preset ratio. The sum of the first preset ratio and the second preset ratio is equal to 1. The elements corresponding to the first number of importance levels are determined as the non-significant elements, and the elements corresponding to the second number of importance levels are determined as the significant elements.

[0103] Optionally, the first bit width is 8 bits and the second bit width is 1 bit.

[0104] Based on the same concept, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any of the above data processing methods.

[0105] Based on the same concept, this disclosure also provides an electronic device that may include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of any of the above data processing methods.

[0106] Based on the same concept, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above data processing methods.

[0107] The following is for reference. Figure 4 This diagram illustrates a structural schematic of an electronic device 400 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0108] like Figure 4 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. Processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0109] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0110] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0111] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0112] In some implementations, communication can be conducted using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0114] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine a target weight matrix for a network layer in a neural network model, and determine the importance of each element in the target weight matrix, wherein the importance of the element is positively correlated with the degree of influence of the element on the model accuracy of the neural network model; determine significant and insignificant elements in the target weight matrix based on the importance, wherein the importance of the significant elements is higher than that of the insignificant elements; quantize the significant elements to a first bit width and quantize the insignificant elements to a second bit width to obtain a weight quantization matrix, wherein the first bit width is greater than the second bit width, and the weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

[0115] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0121] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0122] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A data processing method, characterized in that, The data processing method includes: Determine the target weight matrix of the network layer in the neural network model, and determine the importance of each element in the target weight matrix. The importance of the element is positively correlated with the degree of influence of the element on the model accuracy of the neural network model. Based on the degree of importance, significant and insignificant elements in the target weight matrix are determined, wherein the significance of the significant elements is higher than that of the insignificant elements; The significant elements are quantized to the first bit width, and the non-significant elements are quantized to the second bit width to obtain a weight quantization matrix. The first bit width is greater than the second bit width. The weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

2. The data processing method according to claim 1, characterized in that, Determining the importance of each element in the target weight matrix includes: For each element in the target weight matrix, the element is set to 0 to obtain a weight adjustment matrix. The importance of the element is determined based on the target weight matrix and the weight adjustment matrix.

3. The data processing method according to claim 2, characterized in that, The step of determining the importance of the element based on the target weight matrix and the weight adjustment matrix includes: Determine the first loss value of the network layer in the neural network model using the target weight matrix and the second loss value using the weight adjustment matrix, respectively. Determine the loss difference between the second loss value and the first loss value, and use the loss difference as the importance of the element.

4. The data processing method according to claim 1, characterized in that, The data processing method further includes: Determine the first Hessian matrix of the target weight matrix, and determine the first inverse matrix of the Hessian matrix; Determining the importance of each element in the target weight matrix includes: For each element in the target weight matrix, determine the first element value of the element and determine the second element value of the target element in the first inverse matrix. The square of the first element value is divided by the second element value to obtain the division result as the importance of the element. The target element is the element in the i-th row and i-th column of the first inverse matrix, where i is obtained by multiplying the row number and column number of the element in the target weight matrix and then subtracting 1.

5. The data processing method according to any one of claims 1-4, characterized in that, The step of quantizing the significant elements to the first bit width and the non-significant elements to the second bit width to obtain the weight quantization matrix includes: Use the element corresponding to the first column in the target weight matrix as the target column element; The significant elements in the target column are quantized to the first bit width, and the non-significant elements in the target column are quantized to the second bit width to obtain the quantized column elements; Based on the target weight matrix, the target column elements, and the quantized column elements, a residual compensation matrix is ​​determined. The residual compensation matrix is ​​used to compensate the quantization error between the target column elements and the quantized column elements to the unquantized residual weight matrix in the target weight matrix. The remaining compensation matrix and the remaining weight matrix are added to obtain a new target weight matrix. The element corresponding to the first column of the new target weight matrix is ​​taken as the new target column element. The steps of quantizing the significant elements in the target column element to the first bit width and quantizing the non-significant elements in the target column element to the second bit width are repeated until the last column element in the target weight matrix is ​​quantized to obtain the weight quantization matrix.

6. The data processing method according to claim 5, characterized in that, The step of determining the remaining compensation matrix based on the target weight matrix, the target column elements, and the quantized column elements includes: Determine the second Hessian matrix of the target weight matrix, and determine the second inverse matrix of the Hessian matrix; For each element in the target column, determine the difference between the element and its corresponding quantized element in the quantized column. Divide the difference by the value of the third element of the element in the j-th row and j-th column of the second inverse matrix, and then multiply it by all the elements in the j-th column of the second inverse matrix to obtain the element compensation matrix. Here, j is obtained by multiplying the row number and column number of the element in the target weight matrix and then subtracting 1. The remaining compensation matrix is ​​obtained by summing the element compensation matrices corresponding to all elements in the target column.

7. The data processing method according to any one of claims 1-4, characterized in that, The step of quantizing the significant elements to the first bit width and the non-significant elements to the second bit width to obtain the weight quantization matrix includes: The significant elements are quantized to the first bit width, and the non-significant elements are quantized to the second bit width to obtain the initial quantization matrix; The weighted quantization matrix is ​​obtained by multiplying each insignificant element in the initial quantization matrix by a preset scaling factor of the column containing the insignificant element.

8. The data processing method according to claim 7, characterized in that, The preset scaling factor is determined in the following way: For each column of the target weight matrix, the scaling factor variable that minimizes the L2 norm between the target weight matrix and the target product matrix is ​​determined as the target scaling factor for that column. The target product matrix is ​​obtained by multiplying each insignificant element in the initial quantization matrix by the scaling factor variable of the column containing the insignificant element.

9. The data processing method according to any one of claims 1-4, characterized in that, The step of determining the significant and insignificant elements in the target weight matrix based on the importance level includes: From the importance level, a first number of importance levels and a second number of importance levels are determined, each of the second number of importance levels being higher than the first number of importance levels. The first number is equal to the product of the total number of elements in the target weight matrix and a first preset ratio, and the second number is equal to the product of the total number of elements and a second preset ratio. The sum of the first preset ratio and the second preset ratio is equal to 1. The elements corresponding to the first number of importance levels are determined as the non-significant elements, and the elements corresponding to the second number of importance levels are determined as the significant elements.

10. The data processing method according to any one of claims 1-4, characterized in that, The first bit width is 8 bits, and the second bit width is 1 bit.

11. A data processing apparatus, characterized in that, The data processing device includes: The first determining module is used to determine the target weight matrix of the network layer in the neural network model, and to determine the importance of each element in the target weight matrix. The importance of the element is positively correlated with the degree of influence of the element on the model accuracy of the neural network model. The second determining module is used to determine significant and insignificant elements in the target weight matrix according to the importance level, wherein the importance level of the significant elements is higher than that of the insignificant elements; The quantization module is used to quantize the significant elements to the first bit width and the non-significant elements to the second bit width to obtain a weight quantization matrix. The first bit width is greater than the second bit width. The weight quantization matrix is ​​used to process the input data of the network layer to obtain the output data of the network layer.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the computer program performs the steps of the method described in any one of claims 1-10.

13. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-10.

Images