Weight data processing method of neural network model, electronic equipment and storage medium

Abstract

The invention provides a weight data processing method of a neural network model, electronic equipment and a storage medium, and relates to the technical field of machine learning, and the method comprises the steps: obtaining a weight matrix of a trained neural network model; determining a scaling factor and a zero offset of the weight matrix according to the maximum weight value and the minimum weight value in the weight matrix; based on the scaling factor and the zero offset, performing quantization processing on the weight matrix to obtain a quantized weight matrix; and splitting the quantized weight matrix into a preset number of low-rank matrixes. In the embodiment of the invention, the weight matrix is quantized based on the scaling factor and the zero offset, so that the high precision of the matrix quantization process can be ensured. Through quantization processing and matrix splitting of the weight matrix, dual compression is realized, the compression rate of the weight matrix can be improved, and the storage space of the weight matrix is reduced.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method for processing weight data of a neural network model, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of deep learning and artificial intelligence technologies, the scale and complexity of neural network models are constantly increasing. These models typically contain a large number of weight parameters; for example, Natural Language Processing (NLP) models and Convolutional Neural Network (CNN) models have achieved significant success in various application areas such as computer vision and NLP. However, training and inference of these large-scale models not only require substantial computational resources but also consume significant amounts of storage and memory resources. This poses a significant challenge to deploying deep learning models on resource-constrained devices such as mobile devices and embedded systems.

[0003] To address the applicability of large-scale neural network models in resource-constrained environments, post-training weight compression techniques have emerged. Weight compression aims to reduce the number and accuracy of model parameters through various techniques, thereby lowering storage and computational costs. The challenge lies in achieving efficient weight compression while maintaining data processing accuracy. Summary of the Invention

[0004] This application provides a method for processing weight data of a neural network model, an electronic device, and a storage medium, so as to reduce the storage space of weight data while ensuring the accuracy of data processing.

[0005] In a first aspect, embodiments of this application provide a method for processing weight data of a neural network model, the method comprising:

[0006] Obtain the weight matrix of the trained neural network model; determine the scaling factor and zero offset of the weight matrix based on the maximum and minimum weight values ​​in the weight matrix; quantize the weight matrix based on the scaling factor and zero offset to obtain the quantized weight matrix; split the quantized weight matrix into a preset number of low-rank matrices, where the rank of the low-rank matrices is lower than the rank of the quantized weight matrix.

[0007] Secondly, embodiments of this application provide a method for processing weight data of a neural network model, the method comprising:

[0008] Obtain a predetermined number of low-rank matrices corresponding to the trained neural network model; store these predetermined number of low-rank matrices as the weight matrices of the neural network model. The low-rank matrices are obtained by splitting the quantized weight matrices; the quantized weight matrices are obtained by quantizing the weight matrices of the trained neural network model based on a scaling factor and a zero-point offset; the scaling factor and zero-point offset are determined based on the maximum and minimum weight values ​​in the weight matrix of the neural network model; the rank of the low-rank matrices is lower than the rank of the quantized weight matrices.

[0009] Thirdly, embodiments of this application provide a weight data processing apparatus for a neural network model. The apparatus includes: a matrix acquisition module for acquiring the weight matrix of a trained neural network model; a parameter determination module for determining the scaling factor and zero offset of the weight matrix based on the maximum and minimum weight values ​​in the weight matrix; a quantization processing module for quantizing the weight matrix based on the scaling factor and zero offset to obtain a quantized weight matrix; and a matrix splitting module for splitting the quantized weight matrix into a preset number of low-rank matrices, where the rank of the low-rank matrices is lower than the rank of the quantized weight matrix.

[0010] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods described above when executing the computer program.

[0011] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0012] Sixthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the methods described above.

[0013] Compared with the prior art, this application has the following advantages:

[0014] This application provides a method, electronic device, and storage medium for processing weight data of a neural network model. First, the weight matrix of the trained neural network model is obtained. Then, based on the maximum and minimum weight values ​​in the weight matrix, a scaling factor and a zero-point offset are determined. Based on the scaling factor and the zero-point offset, the weight matrix is ​​quantized to obtain a quantized weight matrix. Finally, the quantized weight matrix is ​​split into a predetermined number of low-rank matrices, where the rank of the low-rank matrices is lower than the rank of the quantized weight matrix. In this embodiment, quantizing the weight matrix based on the scaling factor and the zero-point offset ensures high accuracy in the matrix quantization process. Through weight matrix quantization and matrix splitting, dual compression is achieved, improving the compression ratio of the weight matrix and reducing its storage space.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0017] Figure 1 A flowchart of a weight data processing method for a neural network model provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram illustrating the processing procedure of the weight data processing method for the neural network model provided in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram illustrating the variation of the mean square error between the original weight matrices W and the corresponding restored weight matrices W' provided in the embodiments of this application under different rank parameters.

[0020] Figure 4 This is a schematic diagram illustrating the variation of the compression ratio between the original weight matrices W and the corresponding restored weight matrices W' provided in the embodiments of this application under different rank parameters.

[0021] Figure 5 This is a flowchart of a weight data processing method for a neural network model according to an embodiment of this application.

[0022] Figure 6 This is a flowchart of a weight data processing method for a neural network model according to an embodiment of this application.

[0023] Figure 7 This is a flowchart of a weight data processing method for a neural network model according to an embodiment of this application.

[0024] Figure 8 This is a structural block diagram of the weight data processing device for a neural network model according to an embodiment of this application.

[0025] Figure 9 This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0026] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0027] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0029] This application's technical solution can be applied to scenarios where the weight matrix of a neural network model's parameters is stored after training. Since neural network models have many parameters, and the weight matrix data is large, it consumes significant storage resources. This application quantizes the weight matrix based on scaling factors and zero-point offsets, ensuring high precision in the matrix quantization process. Through weight matrix quantization and matrix splitting, dual compression is achieved, improving the compression ratio of the weight matrix and reducing its storage space. Furthermore, when using the neural network model for inference, dequantization is performed based on the split matrix, scaling factors, and zero-point offsets. The dequantized matrix is ​​then used for neural network model inference, eliminating the need to retrain the neural network model and simplifying the model optimization and deployment process.

[0030] Figure 1 A flowchart of the weight data processing method for the neural network model provided in this application. Figure 2A schematic diagram illustrating the weight data processing method for the neural network model provided in this application. The following is in conjunction with... Figure 1 and Figure 2 The specific processing procedure of this embodiment will be described as follows:

[0031] like Figure 1 As shown, the overall flow of this embodiment is as follows:

[0032] Step S101: Obtain the original weights.

[0033] After the neural network model is trained, the weight matrix of each model parameter is obtained, such as... Figure 2 The weight matrix shown is W. Different model parameters may result in different weight matrices.

[0034] Step S102, Integer asymmetric quantization.

[0035] Integer asymmetric quantization is applied to the weights of the neural network model, converting floating-point numbers into low-precision integers while preserving the non-negativity of the data. This not only reduces the storage requirements for the weights but also leverages the non-negativity of integer data to lay the foundation for subsequent matrix factorization.

[0036] The detailed steps for integer asymmetric quantization are as follows:

[0037] First, determine the maximum (max) and minimum (min) values ​​of the weights in the weight matrix, i.e., the maximum and minimum weight values. Then, use these maximum and minimum weight values ​​to calculate the scaling factor and zero-point offset for each weight matrix. In one example, the formulas for calculating the scaling factor and zero-point offset are as follows:

[0038]

[0039]

[0040] Where scale represents the scaling factor; bits represents the preset quantization precision, for example, the quantization precision can be 8 bits, 4 bits, etc.; zero_ represents the zero offset.

[0041] Then, based on the scaling factor and zero-point offset, the weight matrix is ​​quantized, converting the weight values ​​in the weight matrix into integer representations, such as... Figure 2 The integer asymmetric quantization shown here, in one example, uses the following formula for matrix quantization:

[0042]

[0043] Where W represents the original weight matrix; W q This represents the weight matrix after quantization. This represents the weight matrix after mapping; round() represents the rounding operation.

[0044] Step S103, nonnegative matrix decomposition.

[0045] The quantized weight matrix is ​​split into two low-rank matrices using the Non-negative Matrix Factorization (NMF) algorithm. In one example, the formula for calculating the matrix split is as follows:

[0046] W q ≈A×B(4)

[0047] Where A and B represent the low-rank matrices obtained from the partitioning, and the ranks of A and B are less than W. q .

[0048] After splitting the weight matrix into low-rank matrices, these low-rank matrices are used as the weight matrices of the neural network model and stored, thus reducing storage space. During inference using the neural network model, an approximate weight matrix is ​​reconstructed from the stored low-rank matrices for use in the inference process.

[0049] Step S104: Restore the weight matrix during the inference phase.

[0050] During model inference, the low-rank matrices A and B are recombine as needed to form an approximate matrix W of the quantized weight matrix. q ',like Figure 2 The "low-rank reconstruction" shown here uses a scaling factor and zero-point offset to approximate the quantized weight matrix W. q Perform dequantization. In one example, the formula for dequantization is as follows:

[0051] W′=W' q -zero_point)×scale(5)

[0052] Where W' represents the dequantized weight matrix, which is used for inference calculations in the neural network model.

[0053] In this embodiment, performing integer asymmetric quantization before nonnegative matrix factorization significantly improves the compression ratio. Integer asymmetric quantization converts the floating-point weight matrix into a low-precision integer representation while ensuring the data is nonnegative, providing a better foundation for subsequent matrix factorization. Nonnegative matrix factorization decomposes the quantized integer weight matrix, further reducing the data volume. The combination of these two methods achieves a higher compression ratio while maintaining model performance. Moreover, converting the weight matrix to a nonnegative integer matrix first effectively avoids the complex process of linear transformations required before nonnegative matrix factorization. Integer asymmetric quantization directly constructs input data suitable for nonnegative matrix factorization, enabling smooth nonnegative matrix factorization and thus expanding its applicability. The elimination of additional linear transformations simplifies the implementation process and reduces computational overhead.

[0054] The following section demonstrates the results in terms of mean squared error (MSE) and compression ratio (CR). Figure 3 This is a schematic diagram illustrating the variation of the mean square error between the original weight matrices W and the corresponding restored weight matrices W' under different rank parameters in this embodiment. Figure 3 The horizontal axis represents the rank of the weight matrix, and the vertical axis represents the mean squared error (MSE).

[0055] The formula for calculating the mean squared error is as follows:

[0056]

[0057] Where MSE represents the mean squared error; n represents the number of elements in the weight matrix; y i This represents the values ​​of each element in the restored weight matrix. This represents the corresponding element value in the original weight matrix.

[0058] Figure 4 This is a schematic diagram illustrating the variation of the compression ratio between the original weight matrices W and the corresponding restored weight matrices W' provided in this embodiment under different rank parameters. Figure 4 The horizontal axis represents the rank of the weight matrix, and the vertical axis represents the compression ratio (CR).

[0059] The formula for calculating the compression ratio is as follows:

[0060]

[0061] Where CR represents the compression ratio; Compressed_Size represents the size of the weight matrix after compression using this scheme; and Original_ represents the size of the original weight matrix.

[0062] Combination Figure 3 and Figure 4 In this context, points of different shapes represent weight matrices for different model parameters, i.e., different weight matrices. Figure 3 The comparison of the mean squared errors of the weight matrices for the five model parameters is shown. Figure 4 The comparison of compression ratios for the weight matrices of the four model parameters is shown. From Figure 3 As can be seen, weight matrices of different ranks have different mean square errors after being processed by the weight data processing method in this embodiment. From... Figure 4 As can be seen, weight matrices of different ranks have different compression ratios after being processed by the weight data processing method in this embodiment. Therefore, those skilled in the art can weigh the storage requirements and prediction accuracy of the neural network model when determining the weight matrix compression strategy based on the specific application scenario. In some application scenarios, a higher compression ratio may be needed to save storage space, while in other application scenarios, higher model accuracy may be prioritized, and a lower compression ratio may be acceptable.

[0063] The technical solution of this application will be described below through several embodiments.

[0064] This application provides a method for processing weight data of a neural network model. The method in this embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0065] like Figure 5 The diagram shown is a flowchart of a weight data processing method for a neural network model according to an embodiment of this application, including:

[0066] Step S501: Obtain the weight matrix of the trained neural network model.

[0067] After the neural network model is trained, the weight matrix of each model parameter is obtained. Different model parameters may have different weight matrices. This embodiment will use the weight matrix of any one model parameter as an example.

[0068] Step S502: Determine the scaling factor and zero offset of the weight matrix based on the maximum and minimum weight values ​​in the weight matrix.

[0069] For any weight matrix, where each element represents a weight value, first, calculate the maximum and minimum weight values. Then, using these maximum and minimum weight values, calculate the scaling factor and zero-point offset corresponding to the weight matrix.

[0070] Step S503: Based on the scaling factor and zero offset, the weight matrix is ​​quantized to obtain the quantized weight matrix.

[0071] Among them, quantization of the weight matrix based on scaling factor and zero offset can ensure data processing accuracy.

[0072] Step S504: The quantized weight matrix is ​​split into a preset number of low-rank matrices, the rank of which is lower than the rank of the quantized weight matrix.

[0073] After splitting the weight matrix into low-rank matrices, the low-rank matrices are used as the weight matrices of the neural network model and stored. Since the rank of the low-rank matrix is ​​lower than the rank of the quantized weight matrix, the amount of data stored is reduced.

[0074] The weight data processing method for a neural network model provided in this application first obtains the weight matrix of the trained neural network model; then, based on the maximum and minimum weight values ​​in the weight matrix, it determines the scaling factor and zero-point offset of the weight matrix; based on the scaling factor and zero-point offset, it quantizes the weight matrix to obtain a quantized weight matrix; finally, it splits the quantized weight matrix into a predetermined number of low-rank matrices, where the rank of the low-rank matrices is lower than the rank of the quantized weight matrix. In this application embodiment, quantizing the weight matrix based on the scaling factor and zero-point offset ensures high precision in the matrix quantization process. Through weight matrix quantization and matrix splitting, dual compression is achieved, which improves the compression ratio of the weight matrix and reduces its storage space.

[0075] The following section provides a detailed explanation of the specific implementation process of each of the above steps through various implementation methods.

[0076] In one implementation, step S502, determining the scaling factor and zero offset of the weight matrix based on the maximum and minimum weight values ​​in the weight matrix, includes: step S5021, determining the scaling factor based on the maximum and minimum weight values ​​in the weight matrix; and step S5022, determining the zero offset based on the maximum weight value and the scaling factor.

[0077] The scaling factor is typically used to adjust the range or size of numerical values. It can be calculated using the maximum and minimum weight values, and then used to adjust the weight values ​​to a suitable range or ratio. The zero-point offset is typically used to adjust the starting point or offset of data. It can be used to map the zero point of a floating-point number to the integer range, reducing errors caused by quantization.

[0078] The scaling factor and zero offset are determined and can be used as parameters for quantization of the weight matrix. Then, the weight matrix is ​​quantized based on the scaling factor and zero offset, which can ensure the accuracy of data processing.

[0079] In one implementation, step S5021, determining the scaling factor based on the maximum and minimum weight values ​​in the weight matrix, includes: subtracting the maximum and minimum weight values ​​to obtain a weight difference; and determining the scaling factor based on the weight difference and a preset quantization precision.

[0080] The preset quantization precision can be set according to specific needs; for example, the quantization precision can be 8 bits, 4 bits, etc. Those skilled in the art can set different quantization precisions for different application scenarios.

[0081] In practical applications, the scaling factor can be calculated based on the weight difference between the maximum and minimum weight values ​​and the preset quantization precision. In a specific embodiment, the scaling factor is calculated as shown in formula (1). The scaling factor can be used to adjust the weight values ​​to a suitable range or ratio.

[0082] In one implementation, step S5022, determining the zero-point offset based on the maximum weight value and the scaling factor, includes: performing a division operation on the maximum weight value and the scaling factor to obtain the quotient of the maximum weight value and the scaling factor; and determining the zero-point offset based on the quotient of the maximum weight value and the scaling factor, as well as a preset quantization precision.

[0083] In practical applications, the zero-point offset can be calculated based on the quotient of the maximum weight value and the scaling factor, as well as the preset quantization precision. In a specific embodiment, the zero-point offset is calculated as shown in formula (2). The zero-point offset can be used to adjust or center the distribution of weight values.

[0084] In one implementation, step S503, based on the scaling factor and the zero offset, quantizes the weight matrix to obtain a quantized weight matrix, including: step S5031, based on the scaling factor and the zero offset, mapping each weight value in the weight matrix to obtain mapped weight values; step S5032, rounding each mapped weight value to obtain a quantized weight matrix.

[0085] Among these, you can use rounding functions to perform rounding operations, such as ceil(), floor(), round(), etc.

[0086] In practical applications, the weight values ​​in the weight matrix can be quantized based on the scaling factor and the zero-point offset, i.e., first mapping and then rounding. In a specific embodiment, the calculation method for quantizing the weight matrix is ​​shown in formula (3).

[0087] In this embodiment, quantization of the weight matrix converts the floating-point weight matrix into a low-precision integer representation while ensuring the data is non-negative, providing a foundation for subsequent matrix decomposition. This avoids the complex process of linear transformations required before non-negative matrix decomposition. Integer asymmetric quantization directly constructs input data suitable for non-negative matrix decomposition, enabling smooth non-negative matrix decomposition and thus expanding its applicability. The elimination of additional linear transformations simplifies the implementation process and reduces computational overhead.

[0088] In one implementation, step S504, splitting the quantized weight matrix into a preset number of low-rank matrices, includes: splitting the quantized weight matrix into a first low-rank matrix and a second low-rank matrix, wherein the product of the first low-rank matrix and the second low-rank matrix is ​​an approximate matrix of the quantized weight matrix.

[0089] In practical applications, the non-negative matrix factorization (NMF) algorithm can be used to split the quantized weight matrix into two low-rank matrices. In a specific embodiment, the calculation method for splitting the weight matrix is ​​shown in formula (4).

[0090] In this embodiment, after splitting the weight matrix into low-rank matrices, the low-rank matrices are used as the weight matrices of the neural network model and stored. By splitting the weight matrix, the high-rank matrix can be split into low-rank matrices, thereby reducing the amount of data stored and saving storage space.

[0091] In one implementation, the weight data processing method of the neural network model further includes: obtaining an approximate matrix of the quantized weight matrix based on the first low-rank matrix and the second low-rank matrix; performing dequantization processing on the approximate matrix using a scaling factor and a zero-point offset to obtain the dequantized weight matrix, and using the dequantized weight matrix for inference calculation of the neural network model.

[0092] In practical applications, the stored low-rank matrix is ​​recombined and calculated, and then the approximate matrix obtained by the combined calculation is dequantized using the scaling factor and zero offset to obtain an approximate weight matrix. In a specific embodiment, the dequantization calculation method is shown in formula (5).

[0093] In this embodiment, when using the neural network model for inference, dequantization is performed based on the split matrix, scaling factor, and zero offset. The dequantized matrix is ​​then used as the weight matrix of the neural network model for inference calculations. This eliminates the need to retrain the neural network model, simplifying the model optimization and deployment process.

[0094] In one implementation, the weight data processing method for the neural network model further includes storing a preset number of low-rank matrices as the weight matrices of the neural network model.

[0095] In practical applications, after splitting the weight matrix into multiple low-rank matrices, the low-rank matrices can be stored on the current device as the weight matrices of the neural network model. Since the rank of the low-rank matrix is ​​lower than the rank of the quantized weight matrix, the amount of data stored is reduced compared to directly storing the weight matrix.

[0096] This application provides a method for processing weight data of a neural network model. The method in this embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0097] like Figure 6 The diagram shown is a flowchart of a weight data processing method for a neural network model according to an embodiment of this application, including:

[0098] Step S601: Obtain a preset number of low-rank matrices corresponding to the trained neural network model.

[0099] Step S602: Store a preset number of low-rank matrices as weight matrices for the neural network model.

[0100] The low-rank matrix is ​​obtained by splitting the quantized weight matrix. The quantized weight matrix is ​​obtained by quantizing the weight matrix of the trained neural network model based on the scaling factor and zero offset. The scaling factor and zero offset are determined based on the maximum and minimum weight values ​​in the weight matrix of the neural network model. The rank of the low-rank matrix is ​​lower than the rank of the quantized weight matrix.

[0101] In practical applications, a preset number of low-rank matrices corresponding to a trained neural network model can be received from other devices and stored.

[0102] After the neural network model is trained, the weight matrix for each model parameter is obtained. Different model parameters may result in different weight matrices. For any weight matrix, the elements are weight values. First, the maximum and minimum weight values ​​are calculated, which are the maximum and minimum weight values. Then, using the maximum and minimum weight values, the scaling factor and zero-point offset corresponding to the weight matrix are calculated. Based on the scaling factor and zero-point offset, the weight matrix is ​​quantized, and the quantized weight matrix is ​​split into a predetermined number of low-rank matrices. These low-rank matrices are used as the weight matrix of the neural network model and stored. Since the rank of the low-rank matrices is lower than the rank of the quantized weight matrix, the amount of data stored is reduced.

[0103] When using a neural network model for inference calculations, a preset number of low-rank matrices are combined to obtain an approximate matrix of the quantized weight matrix. The approximate matrix of the quantized weight matrix is ​​then dequantized using a scaling factor and a zero-point offset to obtain the dequantized weight matrix. The dequantized weight matrix is ​​then used for inference calculations of the neural network model.

[0104] It should be noted that the above-mentioned processing of the weight matrix to obtain a preset number of low-rank matrices and the inference calculation using the neural network model can be performed on the same device or on different devices, depending on the specific needs. This application does not impose any restrictions on this.

[0105] In this embodiment, the weight matrix is ​​quantized based on a scaling factor and a zero-point offset, ensuring high precision in the matrix quantization process. Through quantization and matrix splitting of the weight matrix, dual compression is achieved, improving the compression ratio and reducing the storage space of the weight matrix.

[0106] This application provides a method for processing weight data of a neural network model. The method in this embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0107] like Figure 7 The diagram shown is a flowchart of a weight data processing method for a neural network model according to an embodiment of this application, including:

[0108] Step S701: Obtain the weight matrix of the trained neural network model.

[0109] After the neural network model is trained, the weight matrix of each model parameter is obtained. The weight matrix may be different for different model parameters.

[0110] Step S702: Subtract the maximum weight value from the minimum weight value to obtain the weight difference.

[0111] Step S703: Determine the scaling factor based on the weight difference and the preset quantization precision.

[0112] The preset quantization precision can be set according to specific needs; for example, the quantization precision can be 8 bits, 4 bits, etc. Those skilled in the art can set different quantization precisions for different application scenarios. The scaling factor can be used to adjust the weight values ​​to a suitable range or ratio.

[0113] Step S704: Divide the maximum weight value and the scaling factor to obtain the quotient of the maximum weight value and the scaling factor.

[0114] Step S705: Determine the zero offset based on the quotient of the maximum weight value and the scaling factor, as well as the preset quantization precision.

[0115] In practical applications, the zero-point offset can be calculated based on the quotient of the maximum weight value and the scaling factor, as well as the preset quantization precision. Using the zero-point offset, the zero point of a floating-point number can be mapped to the integer range, reducing errors caused by quantization.

[0116] Step S706: Based on the scaling factor and zero offset, the weight values ​​in the weight matrix are mapped to obtain the mapped weight values.

[0117] Step S707: The weight values ​​after mapping are rounded down to obtain the quantized weight matrix.

[0118] Among these, you can use rounding functions to perform rounding operations, such as ceil(), floor(), round(), etc.

[0119] In practical applications, the weight values ​​in the weight matrix can be quantized based on scaling factors and zero-point offsets. Quantization and rounding of the weight matrix converts floating-point weight matrices into low-precision integer representations while ensuring the data is non-negative, providing a foundation for subsequent matrix decomposition. This avoids the complex process of linear transformations required before non-negative matrix decomposition. Integer asymmetric quantization directly constructs input data suitable for non-negative matrix decomposition, enabling smooth decomposition and expanding its applicability. It eliminates the need for additional linear transformations, simplifying the implementation process and reducing computational overhead.

[0120] Step S708: The quantized weight matrix is ​​split into a first low-rank matrix and a second low-rank matrix. The product of the first low-rank matrix and the second low-rank matrix is ​​an approximate matrix of the quantized weight matrix.

[0121] In practical applications, non-negative matrix factorization (NMF) can be used to split the quantized weight matrix into two low-rank matrices. After splitting the weight matrix into low-rank matrices, the low-rank matrices are used as the weight matrices of the neural network model and stored. By splitting the weight matrix, a high-rank matrix can be split into a low-rank matrix, thereby reducing the amount of data stored and saving storage space.

[0122] Step S709: Based on the first low-rank matrix and the second low-rank matrix, obtain an approximate matrix of the quantized weight matrix.

[0123] Step S710: Using the scaling factor and zero offset, the approximate matrix is ​​dequantized to obtain the dequantized weight matrix, which is then used for inference calculation of the neural network model.

[0124] In practical applications, the stored low-rank matrices are recombined and recalculated. Then, the approximate matrix obtained by the combined calculation is dequantized using the scaling factor and zero offset to obtain an approximate weight matrix. When using the neural network model for inference, the dequantized matrix is ​​used as the weight matrix of the neural network model for model inference calculation. This eliminates the need to retrain the neural network model, simplifying the model optimization and deployment process.

[0125] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a weight data processing device for a neural network model. For example... Figure 8 The diagram shown is a structural block diagram of a weight data processing apparatus for a neural network model according to an embodiment of this application. The apparatus includes:

[0126] The matrix acquisition module 801 is used to acquire the weight matrix of the trained neural network model.

[0127] The parameter determination module 802 is used to determine the scaling factor and zero offset of the weight matrix based on the maximum and minimum weight values ​​in the weight matrix.

[0128] The quantization processing module 803 is used to quantize the weight matrix based on the scaling factor and the zero offset to obtain the quantized weight matrix.

[0129] The matrix splitting module 804 is used to split the quantized weight matrix into a preset number of low-rank matrices, the rank of which is lower than the rank of the quantized weight matrix.

[0130] The weight data processing apparatus for a neural network model provided in this application first obtains the weight matrix of the trained neural network model; then, based on the maximum and minimum weight values ​​in the weight matrix, it determines the scaling factor and zero-point offset of the weight matrix; based on the scaling factor and zero-point offset, it quantizes the weight matrix to obtain a quantized weight matrix; finally, it splits the quantized weight matrix into a predetermined number of low-rank matrices, where the rank of the low-rank matrices is lower than the rank of the quantized weight matrix. In this application embodiment, quantizing the weight matrix based on the scaling factor and zero-point offset ensures high precision in the matrix quantization process. Through the quantization and matrix splitting of the weight matrix, dual compression is achieved, which improves the compression ratio of the weight matrix and reduces its storage space.

[0131] In one implementation, the parameter determination module 802 includes a first determination unit and a second determination unit; the first determination unit is used to: determine a scaling factor based on the maximum weight value and the minimum weight value in the weight matrix; the second determination unit is used to: determine a zero-point offset based on the maximum weight value and the scaling factor.

[0132] In one implementation, the first determining unit is specifically used to: perform a subtraction operation between the maximum weight value and the minimum weight value to obtain a weight difference; and determine a scaling factor based on the weight difference and a preset quantization precision.

[0133] In one implementation, the second determining unit is specifically used to: perform a division operation on the maximum weight value and the scaling factor to obtain the quotient of the maximum weight value and the scaling factor; and determine the zero-point offset based on the quotient of the maximum weight value and the scaling factor, as well as the preset quantization precision.

[0134] In one implementation, the quantization processing module 803 is specifically used to: map each weight value in the weight matrix based on the scaling factor and the zero offset to obtain each weight value after mapping; and perform a rounding operation on each weight value after mapping to obtain the quantized weight matrix.

[0135] In one implementation, the matrix splitting module 804 is specifically used to: split the quantized weight matrix into a first low-rank matrix and a second low-rank matrix, the product of the first low-rank matrix and the second low-rank matrix being an approximate matrix of the quantized weight matrix.

[0136] In one implementation, the weight data processing device of the neural network model further includes an inverse quantization module; the inverse quantization module is used to: obtain an approximate matrix of the quantized weight matrix based on the first low-rank matrix and the second low-rank matrix; and perform inverse quantization processing on the approximate matrix using a scaling factor and a zero-point offset to obtain the inverse quantized weight matrix, which is used for inference calculation of the neural network model.

[0137] In one implementation, the weight data processing device of the neural network model is further used to: store a preset number of low-rank matrices as weight matrices of the neural network model.

[0138] The functions of each module in the embodiments of this application can be found in the corresponding descriptions in the above methods, and they have corresponding beneficial effects, which will not be repeated here.

[0139] Figure 9 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 9 As shown, the electronic device includes a memory 910 and a processor 920. The memory 910 stores a computer program that can run on the processor 920. When the processor 920 executes the computer program, it implements the method described in the above embodiments. The number of memories 910 and processors 920 can be one or more.

[0140] The electronic device also includes:

[0141] The communication interface 930 is used to communicate with external devices and exchange and transmit data.

[0142] If the memory 910, processor 920, and communication interface 930 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0143] Optionally, in a specific implementation, if the memory 910, processor 920, and communication interface 930 are integrated on a single chip, then the memory 910, processor 920, and communication interface 930 can communicate with each other through an internal interface.

[0144] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0145] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the method provided in this application.

[0146] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0147] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0148] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0149] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0150] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0151] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0152] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0153] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0154] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0155] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0157] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing weight data in a neural network model, characterized in that, The method includes: Obtain the weight matrix of the trained neural network model; Based on the maximum and minimum weight values ​​in the weight matrix, determine the scaling factor and zero offset of the weight matrix; Based on the scaling factor and the zero offset, the weight matrix is ​​quantized to obtain the quantized weight matrix. The quantized weight matrix is ​​split into a predetermined number of low-rank matrices, wherein the rank of the low-rank matrices is lower than the rank of the quantized weight matrix.

2. The method according to claim 1, characterized in that, The step of determining the scaling factor and zero-point offset of the weight matrix based on the maximum and minimum weight values ​​in the weight matrix includes: The scaling factor is determined based on the maximum and minimum weight values ​​in the weight matrix; The zero-point offset is determined based on the maximum weight value and the scaling factor.

3. The method according to claim 2, characterized in that, Determining the scaling factor based on the maximum and minimum weight values ​​in the weight matrix includes: The weight difference is obtained by subtracting the maximum weight value from the minimum weight value. The scaling factor is determined based on the weight difference and the preset quantization precision.

4. The method according to claim 2, characterized in that, Determining the zero-point offset based on the maximum weight value and the scaling factor includes: Divide the maximum weight value and the scaling factor to obtain the quotient of the maximum weight value and the scaling factor; The zero-point offset is determined based on the quotient of the maximum weight value and the scaling factor, as well as the preset quantization precision.

5. The method according to any one of claims 1-4, characterized in that, The step of quantizing the weight matrix based on the scaling factor and the zero-point offset to obtain the quantized weight matrix includes: Based on the scaling factor and the zero offset, the weight values ​​in the weight matrix are mapped to obtain the mapped weight values. The weight values ​​after the mapping process are rounded down to obtain the quantized weight matrix.

6. The method according to any one of claims 1-4, characterized in that, The step of splitting the quantized weight matrix into a predetermined number of low-rank matrices includes: The quantized weight matrix is ​​split into a first low-rank matrix and a second low-rank matrix, and the product of the first low-rank matrix and the second low-rank matrix is ​​an approximation of the quantized weight matrix.

7. The method according to claim 6, characterized in that, The method further includes: Based on the first low-rank matrix and the second low-rank matrix, an approximate matrix of the quantized weight matrix is ​​obtained; The approximate matrix is ​​dequantized using the scaling factor and the zero offset to obtain the dequantized weight matrix, which is then used for inference calculations of the neural network model.

8. The method according to claim 1, characterized in that, The method further includes: The predetermined number of low-rank matrices are stored as the weight matrices of the neural network model.

9. A method for processing weight data in a neural network model, characterized in that, The method includes: Obtain a predetermined number of low-rank matrices corresponding to the trained neural network model; The predetermined number of low-rank matrices are stored as the weight matrices of the neural network model. The low-rank matrix is ​​obtained by splitting the quantized weight matrix; the quantized weight matrix is ​​obtained by quantizing the weight matrix of the trained neural network model based on a scaling factor and a zero-point offset; the scaling factor and the zero-point offset are determined based on the maximum and minimum weight values ​​in the weight matrix of the neural network model; the rank of the low-rank matrix is ​​lower than the rank of the quantized weight matrix.

10. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images