Model processing method and device, equipment and storage medium

Abstract

The invention provides a model processing method and device, equipment and a storage medium, and relates to the field of data processing, in particular to the fields of artificial intelligence, big data and the like. According to the specific implementation scheme, a target model is determined, and the target model is a trained neural network model; based on the distribution characteristics of the weight parameters in the target model, performing quantization processing on the weight parameters in the target model to obtain weight indexes corresponding to the weight parameters; wherein the storage space occupied by the weight index corresponding to the weight parameter is smaller than the storage space occupied by the weight parameter; and storing the weight index corresponding to each weight parameter so as to reduce the storage space occupied by the target model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and in particular to the technical field of artificial intelligence, big data, etc. BACKGROUND

[0002] With the rapid development of deep learning technology, large models are becoming more and more complex and huge, usually containing millions of parameters, which in turn leads to huge demand for computing resources and storage space of large models. In order to reduce the computing and storage demand of large models, large model quantization technology emerges as the times require. However, the existing quantization methods still have defects in storage demand and model performance, therefore, there is an urgent need for a new large model quantization scheme. SUMMARY

[0003] The present disclosure provides a model processing method, apparatus, device and storage medium.

[0004] According to an aspect of the present disclosure, a model processing method is provided, comprising:

[0005] determining a target model, wherein the target model is a trained neural network model;

[0006] based on the distribution characteristics of the weight parameters in the target model, performing quantization processing on each weight parameter in the target model to obtain a weight index corresponding to each weight parameter; wherein the storage space occupied by the weight index corresponding to the weight parameter is less than the storage space occupied by the weight parameter;

[0007] storing the weight index corresponding to each weight parameter to reduce the storage space occupied by the target model.

[0008] According to another aspect of the present disclosure, a model processing apparatus is provided, comprising:

[0009] a model processing unit configured to determine a target model, wherein the target model is a trained neural network model; based on the distribution characteristics of the weight parameters in the target model, perform quantization processing on each weight parameter in the target model to obtain a weight index corresponding to each weight parameter; wherein the storage space occupied by the weight index corresponding to the weight parameter is less than the storage space occupied by the weight parameter;

[0010] a storage unit configured to store the weight index corresponding to each weight parameter to reduce the storage space occupied by the target model.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected with the at least one processor; wherein

[0014] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method according to any of the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0017] In this way, the scheme of the present disclosure can quantize the weight parameters according to the distribution characteristics of the weight parameters in the target model, and obtain weight indexes of the weight parameters which occupy less storage space (for example, compared with the weight parameters, the weight indexes can be represented by low bits), so that the storage requirement of the target model is significantly reduced, and the deployment efficiency of the target model in a resource-limited environment is improved.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0020] Figure 1 is a schematic flow of a model processing method according to an embodiment of the present application Figure 1 ;

[0021] Figure 2 is an effect diagram of the distribution of the normalized weight parameters according to an embodiment of the present application

[0022] Figure 3 is a schematic flow of a model processing method according to an embodiment of the present application Figure 2 ;

[0023] Figure 4 is a distribution diagram of a preset sequence matched with the non-uniform distribution characteristics of the weight parameters according to an embodiment of the present application

[0024] Figure 5 is a quantization processing flow diagram of the weight parameters in a specific example of the model processing method according to an embodiment of the present application

[0025] FIG. 6(a) and FIG. 6(b) are schematic diagrams of inference processes of a target model according to another embodiment of the present application;

[0026] Figure 7 is a structural schematic diagram of a model processing apparatus according to an embodiment of the present application;

[0027] Figure 8 is a block diagram of an electronic device for implementing a model processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, in which various specific details are set forth to assist in a thorough understanding of the embodiments of the present application. It will be understood by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Also, the description is made in the order of the following description for clarity and conciseness.

[0029] The term "and / or", used herein only describes an associated relationship for associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. The term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C, which means including any one or more elements selected from the set consisting of A, B, and C. The terms "first", "second", etc. herein refer to a plurality of similar technical terms and distinguish them, and do not mean the order or mean only two, for example, the first feature and the second feature refer to two categories / two features, the first feature can be one or more, and the second feature can also be one or more.

[0030] In addition, in order to better illustrate the present application, a large number of specific details are given in the following detailed description. Those skilled in the art should understand that the present application can also be implemented without some specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail, in order to highlight the main idea of the present application.

[0031] The related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way as optional schemes, which all belong to the protection scope of the embodiments of the present application.

[0032] With the rapid development of deep learning technology, large models are becoming more and more complex and huge, usually containing millions of parameters, resulting in huge demand for computing resources and storage space of large models. In order to reduce the computing and storage requirements of large models, large model quantization technology emerges as the times require. However, the existing quantization methods still have defects in storage requirements and model performance, therefore, there is an urgent need for a new large model quantization scheme.

[0033] Based on this, the present scheme proposes a model processing method to enable large models to reduce the storage requirements of the model while maintaining the performance of the model.

[0034] Specifically, Figure 1 is a schematic flow of the model processing method according to an embodiment of the present application Figure 1 . The method can be optionally applied in electronic devices such as personal computers, servers, server clusters, etc.

[0035] Further, the method includes at least part of the following content. As shown in Figure 1 , it includes:

[0036] Step S101: The processor determines a target model.

[0037] Here, the target model is a trained neural network model. For example, the target model is a large language model, etc., and the present scheme does not limit the specific model.

[0038] Step S102: The processor quantizes each weight parameter in the target model based on the distribution characteristics of the weight parameters in the target model to obtain the weight index corresponding to each weight parameter.

[0039] Here, the weight index corresponding to the weight parameter occupies less storage space than the weight parameter. In this way, the storage space occupied by the quantized target model is effectively reduced.

[0040] Step S103: The memory stores the weight index corresponding to each weight parameter to reduce the storage space occupied by the target model.

[0041] For example, when storing the target model, the memory stores the weight index corresponding to each weight parameter instead of the specific weight parameter, so as to reduce the storage space occupied by the target model.

[0042] In other words, after model training, the processor can quantize the weight parameters based on the distribution characteristics of the weight parameters in the target model obtained after training, and obtain the index of the weight parameters. Then, when the memory stores the target model, it does not need to store the weight parameters that occupy a lot of space, but stores the index corresponding to the weight parameters. In this way, the storage space required for model storage is effectively reduced.

[0043] In this way, the disclosed solution can quantify the weight parameters according to the distribution characteristics of the weight parameters in the target model and obtain the weight index of each weight parameter with a small storage space (for example, the weight index can be represented by low bits compared to the weight parameters). This significantly reduces the storage requirements of the target model and improves the deployment efficiency of the target model in resource-constrained environments.

[0044] Moreover, the quantization process of this disclosed solution fully considers the distribution characteristics of the weight parameters, thus ensuring the lossless nature of the weight information during quantization to the greatest extent possible. This lays the foundation for maintaining high model performance after quantization and also lays the foundation for efficient model inference and improved user experience.

[0045] In addition, since the quantization process of this disclosure fully considers the distribution characteristics of the weight parameters, compared with the existing integer quantization method (integer quantization is beneficial for hardware acceleration and reducing storage requirements, but the quantization process adopts a uniformity rule, which is not conducive to maintaining the accuracy of the model), it effectively avoids the accumulation of quantization error and lays the foundation for ensuring the inference performance of the model.

[0046] It should be noted that in practical scenarios, the weight parameters in the target model can be represented using single-precision floating-point numbers (float32) or double-precision floating-point numbers (float64). These data types require 32 bits or 64 bits of storage space, resulting in high storage requirements for storing the target model. The present invention can quantize the weight parameters represented by high bits (e.g., 32 bits or 16 bits) to obtain weight indices that can be represented by low bits (e.g., 4 bits), and store the target model by storing the weight indices represented by low bits. Thus, compared to directly storing the weight parameters by high bits, the present invention effectively reduces the storage space occupied by the target model, reduces the storage cost required by the target model, and achieves effective compression of the target model.

[0047] In a specific example, before quantizing the weight parameters in the target model, preprocessing is required. For instance, firstly, the weight parameters are normalized to ensure their values ​​fall within the range of [-1, 1]. Then, a percentage truncation method is used to truncate the normalized weight parameters to remove extreme values ​​(or outliers). For example, values ​​whose absolute values ​​are far from the target value (e.g., 0) are removed. Figure 2 As shown, the normalized weight parameters are truncated using the 99.99% quantile, resulting in the following: Figure 2 The distribution characteristics of the weight parameters are shown. Here, since the above processing can quickly remove weight parameters with large values ​​and small numbers (i.e., extreme values), it can effectively avoid weighting loss.

[0048] Figure 3 This is an illustrative flow diagram of a model processing method according to an embodiment of this application. Figure 2 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that the above... Figure 1 and Figure 2 The methods shown can also be applied to this example, but the related content will not be elaborated further in this example.

[0049] Furthermore, the method includes at least a portion of the following: For example... Figure 3 As shown, the method includes:

[0050] Step S301: Determine the target model.

[0051] Here, the target model is a trained neural network model.

[0052] Step S302: Determine a preset sequence that matches the distribution characteristics of the weight parameters in the target model.

[0053] Here, the length of the preset sequence is related to the preset quantization requirements. Furthermore, the length of the preset sequence (which can be denoted as L) can be obtained based on the quantization precision (e.g., the number of bits b) required to quantize the weight parameters in the target model. For example, the length of the preset sequence L = 2. b In this case, the weight index can be represented by b bits; for example, in one example, the weight index obtained by the present invention can be represented by 4 bits, which can significantly reduce the storage space required to store the target model.

[0054] In addition, in this example, in order to reduce information loss during the quantization process and avoid the accumulation of accuracy loss during the quantization process, the present disclosure scheme pre-sets the data distribution characteristics of the sequence to match the distribution characteristics of the weight parameters. For example, the data distribution of the pre-set sequence can best characterize the distribution characteristics of the weight parameters. For instance, when the weight parameters have non-uniform and symmetrical distribution characteristics, the data distribution of the pre-set sequence also has non-uniform and symmetrical characteristics, thereby reducing information loss during the quantization process.

[0055] For example, if it is determined that the weight parameters of the target model need to be quantized into 4-bit values ​​(corresponding to weight indices), the non-uniform distribution characteristics of the weight parameters in the target model (e.g., the weight parameter values ​​are densely concentrated near the value 0 and sparse elsewhere) are used to determine the weight index. Figure 4 The preset sequence c shown matches the non-uniform distribution characteristics of the weight parameters in the target model. (4) That is, c (4) [0:16] = [0.0, 0.03125, 0.1015625, 0.171875, 0.2578125, 0.3515625, 0.5, 0.9921875, -0.0078125, -0.0390625, -0.109375, -0.1796875, -0.265625, -0.3671875, -0.5078125, -1.0], where the preset sequence c (4) Length L = 2 4 =16.

[0056] Step S303: At least using a preset sequence that matches the distribution characteristics of the weight parameters in the target model, quantize each weight parameter in the target model to obtain the weight index corresponding to each weight parameter.

[0057] Here, the storage space occupied by the weight index corresponding to the weight parameter is less than the storage space occupied by the weight parameter itself. For example, in one example, the weight index obtained by the present invention can be represented by 4 bits, which significantly reduces the storage space occupied compared to the weight parameters represented by 32 bits or 16 bits.

[0058] Step S304: Store the weight index corresponding to each weight parameter to reduce the storage space occupied by the target model.

[0059] Thus, the present invention can obtain a preset sequence that matches the distribution characteristics of the weight parameters based on the distribution characteristics of the weight parameters, and then use the obtained preset sequence to quantize each weight parameter. Here, since the preset sequence is obtained by fully considering the distribution characteristics of the weight parameters, that is, the preset sequence can characterize the distribution characteristics of the weight parameters, the present invention can effectively reduce information loss during the quantization process and avoid the accumulation of accuracy loss during the quantization process. At the same time, while reducing the storage space and computing resources required by the target model, it achieves effective compression of the target model, making the target model easier to deploy in resource-constrained environments.

[0060] Further, in a specific example, the weight index corresponding to each weight parameter can be obtained in the following manner; specifically, the above-mentioned method of at least using a preset sequence that matches the distribution characteristics of the weight parameters in the target model to quantize each weight parameter in the target model to obtain the weight index corresponding to each weight parameter (for example, step S303 mentioned above) specifically includes:

[0061] Step S303-1: Obtain the extreme values ​​of the weights based on the weight parameters in the target model.

[0062] For example, in one example, the extreme value of the weight could be the maximum value among the weight parameters.

[0063] Step S303-2: Based on the relationship between the extreme values ​​of the weights, the weight parameters in the target model, and the values ​​in the preset sequence, quantize each weight parameter in the target model to obtain the weight index corresponding to each weight parameter.

[0064] Thus, the present disclosure provides a refinement scheme that uses weight extrema, weight parameters, and preset sequences to perform non-uniform quantization processing on each weight parameter. In this way, a weight index that can be represented using low bits can be obtained, which effectively reduces the storage space required by the target model and achieves effective compression of the target model. Moreover, the processing is simple and efficient, and also effectively reduces the required computing resources, making the target model easier to deploy in resource-constrained environments.

[0065] Furthermore, in one example, the weight index corresponding to each weight parameter can be obtained by the following formula:

[0066]

[0067] Here, W ij This represents the ij-th weight parameter in the target model. Represents the weight parameter W ij The corresponding weight index, |W| maxThis represents the extreme value of the weights; for example, in this case, it represents the maximum value among the weight parameters. Represents the preset sequence c (b) The l-th value in the sequence is b, which represents the number of bits required to store the weight index (that is, the weight index can be represented and stored by b bits). The value of l is related to the length L of the preset sequence. For example, the value range of l is [0, L-1].

[0068] It should be noted that the weight parameters in this example can also be represented by a weight matrix, in which case W ij This can represent the weight parameter in the i-th row and j-th column of the weight matrix W in the target model, |W| max This is the maximum value of all elements in the weight matrix W.

[0069] Thus, the present disclosure provides a specific scheme for the quantization of weight parameters, which simplifies the quantization process of weight parameters in the target model, effectively reduces the computational complexity of the quantization process, and thus effectively reduces the required computational resources.

[0070] It should be noted that, in one example, before quantizing the weight parameters in the target model, an equivalent transformation operation can be performed on the weight parameters and activation parameters in the target model. This provides a foundation for the matrix multiplication of weight parameters and activation parameters in the subsequent model inference stage, as well as for dequantization. Furthermore, as... Figure 5 As shown, first, the weight parameters (W) and activation parameters (X) in the target model are subjected to equivalent transformation operations. Then, the weight parameters (which can be denoted as W_s) after the equivalent transformation operation are subjected to the non-uniform quantization process described above to obtain the weight index represented by low bits (e.g., 4 bits).

[0071] Figures 6(a) and 6(b) are schematic diagrams of the reasoning process of the target model according to another embodiment of this application; further, in a specific example, the model processing method may further include the following specific components during the model reasoning stage:

[0072] Step S601: When performing model inference using the target model, determine new weight parameters that match the weight index.

[0073] Here, the new weight parameters occupy more storage space than the weight index, thus laying the foundation for ensuring inference accuracy.

[0074] For example, in one instance, when the weight indices are stored in 4-bit representations and model inference is performed using the target model, the stored 4-bit weight indices can be converted into 8-bit weight parameters to obtain weight parameters that meet the requirements of model inference. This effectively ensures that the model performance is not affected; in other words, it effectively ensures that the inference accuracy is not affected by the quantization process.

[0075] Step S602: Perform model inference based on the new weight parameters that match the weight index.

[0076] In this way, the proposed solution can effectively ensure that the model performance is not affected while reducing the storage space and computing resources required for the target model.

[0077] It should be noted that, for large language models, the model processing method provided in this disclosure enables large language models to perform inference operations efficiently and accurately in the cloud, so as to quickly respond to user requests. In this way, the overall service efficiency and response speed are improved, and the user experience is effectively enhanced.

[0078] In one example, the new weight parameters can be obtained as follows; specifically, the determination of new weight parameters matching the weight index as described above (e.g., step S601 described above), as shown in Figure 6(b), specifically includes:

[0079] Based on the stored preset index table, determine the weight index (e.g., the weight index W represented by 4 bits). q_4 A new weight parameter that matches (e.g., a new weight parameter W represented in 8 bits). q_8 This is used to convert the weight index into weight parameters required for model inference.

[0080] It should be noted that the elements in the preset index table are related to the preset sequence. Furthermore, the preset index table is obtained by processing the preset sequence using preset conversion relationships. Here, preset conversion relationships refer to the conversion relationships between different bits, and further refer to the conversion relationships between the bits representing the parameter index and the bits representing the new weight parameters.

[0081] For example, the weight index is represented by 4 bits, and the new weight parameters need to be represented by 8 bits. Furthermore, the preset sequence c... (4)[0:16] = [0.0, 0.03125, 0.1015625, 0.171875, 0.2578125, 0.3515625, 0.5, 0.9921875, -0.0078125, -0.0390625, -0.109375, -0.1796875, -0.265625, -0.3671875, -0.5078125, -1.0]. At this point, the preset index table c can be obtained. (8) That is, c (8) [0:16] = [0,4,13,22,33,45,64,127,-1,-5,-14,-23,-34,-46,-65,-128].

[0082] It should be noted that since the preset index table is based on a preset sequence, the distribution characteristics of the new weight parameters are matched with the distribution characteristics of the weight parameters in the target model before quantization, thus effectively ensuring the performance of the model.

[0083] Furthermore, in one example, the weight index is represented by b bits. The new matching weight parameters are obtained using the following formula:

[0084]

[0085] here, Representation and weight index Matching new weight parameters; c (b′) represents the preset index table, b represents the number of bits required to store the weight index, and b′ represents the number of bits required to represent the new weight parameters during inference. In other words, the new weight parameters are represented by b′ bits.

[0086] For example, in a weight index that needs to be represented by 4 bits Convert the new weight parameters to 8-bit representation, with the preset index table being c. (8) In this case, based on the stored preset index table c (8) It can quickly determine the weight index represented by 4 bits. Matching new weight parameters

[0087] It should be noted that, in order to achieve a faster response speed during the model inference stage, in this example, after quantization, the memory can also store a preset index table. Moreover, the storage space occupied by this preset index table is much smaller than the storage space required for the weight parameters in the target model. Thus, during the model inference stage, the processor can quickly restore the weight index represented by low bits into weight parameters that meet the requirements of model inference. Compared with directly storing the values ​​of weight parameters, the present solution can effectively reduce the storage space occupied. On the other hand, because a preset index table is stored and new weight parameters carrying more weight information corresponding to the weight index can be quickly found, the solution effectively reduces the storage space occupied and also effectively avoids accuracy loss, thus ensuring the performance of the model.

[0088] In this way, the proposed solution can quickly determine new weight parameters that match the weight index by using a pre-set index table for model inference. This effectively reduces the storage space required and avoids accuracy loss, thus ensuring model performance.

[0089] In a specific example, model inference can be performed in the following manner; specifically, the model inference based on new weight parameters that match the weight index (e.g., step S602 above), as shown in Figure 6(b), specifically includes:

[0090] Based on new weight parameters that match the weight index (e.g., new weight parameters W represented in 8 bits). q_8 ), and the quantized activation parameters in the target model (e.g., activation parameters X represented by 8 bits). q_8 The target model is dequantized to perform model inference.

[0091] In other words, the disclosed solution performs dequantization on the new weight parameters and the quantized activation parameters during the model inference stage to restore the accuracy of the model's weight parameters and activation parameters, thereby enabling the model to maintain its original performance as much as possible during the inference process.

[0092] In a specific example of the scheme disclosed herein, the quantized activation parameters can be obtained in the following manner; specifically, before performing dequantization based on new weight parameters matching the weight index and the quantized activation parameters in the target model, the activation parameters can be quantized in the following manner, as shown in Figure 6(b), including:

[0093] Obtain the stored size factor; quantize the activation parameters in the target model based on the size factor (e.g., denoted as S_x) (e.g., perform static integer quantization) to obtain the quantized activation parameters (e.g., activation parameters X represented by 8 bits). q_8 ).

[0094] Here, the size factor is pre-stored. Further, in one example, the size factor may be based on a calibration dataset, which is derived from statistical information (such as maximum, minimum, and mean values) of the activation parameters in the target model.

[0095] In other words, in order to align the activation parameters represented by high bits (e.g., 32 bits) in the target model with the new weight parameters (e.g., with the new weight parameters represented by 8 bits) in terms of precision, the activation parameters in the target model can be quantized using a pre-stored size factor to obtain quantized activation parameters that are aligned with the precision of the new weight parameters (e.g., the new weights represented by 8 bits). This facilitates better subsequent fusion processing with the new weight parameters, thereby effectively improving the computational efficiency during model inference while ensuring inference accuracy.

[0096] It should be noted that the activation parameters can be quantized during the inference stage to obtain low-bit representations of the activation parameters, which can then be temporarily stored and released after inference. This effectively reduces storage space while ensuring inference performance.

[0097] Furthermore, in a specific example, the above-described inverse quantization process, based on new weight parameters matching the weight index and the quantized activation parameters, as shown in Figure 6(b), can specifically include:

[0098] The new weight parameters (e.g., activation parameters W represented in 8 bits) will be matched with the weight index. q_8 ), and the quantized activation parameters (e.g., activation parameters X represented by 8 bits). q_8 The fusion process is performed to perform inverse quantization based on the fusion result (e.g., a matrix represented by 32 bits).

[0099] In one example, the new weight parameters can be represented by a weight matrix. Similarly, the quantized activation parameters can also be represented by an activation matrix. In this case, the process of fusing the new weight parameters matched with the weight index and the quantized activation parameters can specifically include: performing matrix multiplication between the new weight matrix matched with the weight index and the quantized activation matrix, and then performing inverse quantization on the result. This improves computational efficiency during model inference while maintaining inference accuracy.

[0100] This disclosure also provides a model processing device, such as... Figure 7 As shown, it includes:

[0101] The model processing unit 701 is used to determine the target model, wherein the target model is a trained neural network model; based on the distribution characteristics of the weight parameters in the target model, the weight parameters in the target model are quantized to obtain the weight index corresponding to each weight parameter; wherein the storage space occupied by the weight index corresponding to the weight parameter is less than the storage space occupied by the weight parameter.

[0102] Storage unit 702 is used to store the weight index corresponding to each weight parameter, so as to reduce the storage space occupied by the target model.

[0103] In a specific example of the disclosed solution, the model processing unit is specifically used for:

[0104] Determine a preset sequence that matches the distribution characteristics of the weight parameters in the target model;

[0105] At least one preset sequence matching the distribution characteristics of the weight parameters in the target model is used to quantize each weight parameter in the target model in order to obtain the weight index corresponding to each weight parameter.

[0106] In a specific example of the disclosed solution, the model processing unit is specifically used for:

[0107] Based on the weight parameters in the target model, the extreme values ​​of the weights are obtained;

[0108] Based on the relationship between the extreme values ​​of the weights, the weight parameters in the target model, and the values ​​in the preset sequence, the weight parameters in the target model are quantized to obtain the weight index corresponding to each weight parameter.

[0109] In a specific example of the scheme disclosed herein, the weight index corresponding to each weight parameter is obtained by the following formula:

[0110]

[0111] Among them, W ij This represents the ij-th weight parameter in the target model. Represents the weight parameter W ij The corresponding weight index, |W| max Indicates the extreme value of the weight. Represents the preset sequence c (b) The l-th value in the sequence, b represents the number of bits required to store the weight index, and the value of l is related to the preset sequence c. (b) It is related to the length L.

[0112] In a specific example of the scheme disclosed herein, the model processing unit is further configured to:

[0113] When performing model inference using the target model, new weight parameters that match the weight index are determined; the storage space occupied by the new weight parameters is greater than the storage space occupied by the weight index.

[0114] Model inference is performed based on new weight parameters that match the weight index.

[0115] In a specific example of the scheme disclosed herein,

[0116] The storage unit is also used to store a preset index table;

[0117] The model processing unit is specifically used to determine new weight parameters that match the weight index based on a stored preset index table, so as to convert the weight index into weight parameters required for model inference.

[0118] In a specific example of the disclosed solution, the model processing unit is specifically used for:

[0119] Based on the new weight parameters that match the weight index and the quantized activation parameters in the target model, dequantization is performed to utilize the dequantized target model for model inference.

[0120] In a specific example of the scheme disclosed herein,

[0121] The storage unit is also used to store size factors;

[0122] The model processing unit is also used to acquire the stored size factor; and to quantize the activation parameters in the target model based on the size factor to obtain the quantized activation parameters.

[0123] In a specific example of the disclosed solution, the model processing unit is specifically used for:

[0124] The new weight parameters, which match the weight index, are fused with the quantized activation parameters to perform inverse quantization based on the fusion result.

[0125] For a description of the specific functions and examples of each unit of the apparatus in this disclosure embodiment, please refer to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be repeated here.

[0126] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0127] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0128] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0130] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as model processing methods. For example, in some embodiments, the model processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the model processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform model processing methods by any other suitable means (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0137] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0138] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0139] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model processing method, comprising: Determine the target model, wherein the target model is a trained neural network model; Based on the distribution characteristics of the weight parameters in the target model, each weight parameter in the target model is quantized to obtain the weight index corresponding to each weight parameter; wherein, the storage space occupied by the weight index corresponding to the weight parameter is less than the storage space occupied by the weight parameter. Store the weight index corresponding to each weight parameter to reduce the storage space occupied by the target model.

2. The method according to claim 1, wherein, The step of quantizing each weight parameter in the target model based on the distribution characteristics of the weight parameters in the target model to obtain the weight index corresponding to each weight parameter includes: Determine a preset sequence that matches the distribution characteristics of the weight parameters in the target model; At least one preset sequence matching the distribution characteristics of the weight parameters in the target model is used to quantize each weight parameter in the target model in order to obtain the weight index corresponding to each weight parameter.

3. The method according to claim 2, wherein, The step of quantizing each weight parameter in the target model using at least a preset sequence that matches the distribution characteristics of the weight parameters in the target model to obtain the weight index corresponding to each weight parameter includes: Based on the weight parameters in the target model, the extreme values ​​of the weights are obtained; Based on the relationship between the extreme values ​​of the weights, the weight parameters in the target model, and the values ​​in the preset sequence, the weight parameters in the target model are quantized to obtain the weight index corresponding to each weight parameter.

4. The method according to claim 3, wherein, The weight index corresponding to each weight parameter is obtained by the following formula: Among them, W ij This represents the ij-th weight parameter in the target model. Represents the weight parameter W ij The corresponding weight index, |W| max Indicates the extreme value of the weight. Represents the preset sequence c (b) The l-th value in the sequence, b represents the number of bits required to store the weight index, and the value of l is related to the preset sequence c. (b) It is related to the length L.

5. The method according to any one of claims 1-4, further comprising: When using the target model for model inference, new weight parameters that match the weight index are determined; The new weight parameters occupy more storage space than the weight index. Model inference is performed based on new weight parameters that match the weight index.

6. The method according to claim 5, wherein, The process of determining new weight parameters that match the weight index includes: Based on a stored preset index table, new weight parameters that match the weight index are determined to convert the weight index into weight parameters required for model inference.

7. The method according to claim 5, wherein, The model inference based on new weight parameters that match the weight index includes: Based on the new weight parameters that match the weight index and the quantized activation parameters in the target model, dequantization is performed to utilize the dequantized target model for model inference.

8. The method according to claim 7, wherein, Before performing dequantization based on new weight parameters matched with the weight index and the quantized activation parameters in the target model, the method further includes: Get the storage size factor; The activation parameters in the target model are quantized based on the size factor to obtain the quantized activation parameters.

9. The method according to claim 7, wherein, The process of inverse quantizing the activation parameters, based on new weight parameters that match the weight index, includes: The new weight parameters, which match the weight index, are fused with the quantized activation parameters to perform inverse quantization based on the fusion result.

10. A model processing apparatus, comprising: A model processing unit is used to determine a target model, wherein the target model is a trained neural network model; based on the distribution characteristics of the weight parameters in the target model, the unit quantizes each weight parameter in the target model to obtain the weight index corresponding to each weight parameter; wherein the storage space occupied by the weight index corresponding to the weight parameter is less than the storage space occupied by the weight parameter. The storage unit is used to store the weight indexes corresponding to each weight parameter, so as to reduce the storage space occupied by the target model.

11. The apparatus according to claim 10, wherein, The model processing unit is specifically used for: Determine a preset sequence that matches the distribution characteristics of the weight parameters in the target model; At least one preset sequence matching the distribution characteristics of the weight parameters in the target model is used to quantize each weight parameter in the target model in order to obtain the weight index corresponding to each weight parameter.

12. The apparatus according to claim 11, wherein, The model processing unit is specifically used for: Based on the weight parameters in the target model, the extreme values ​​of the weights are obtained; Based on the relationship between the extreme values ​​of the weights, the weight parameters in the target model, and the values ​​in the preset sequence, the weight parameters in the target model are quantized to obtain the weight index corresponding to each weight parameter.

13. The apparatus according to claim 12, wherein, The weight index corresponding to each weight parameter is obtained by the following formula: Among them, W ij This represents the ij-th weight parameter in the target model. Represents the weight parameter W ij The corresponding weight index, |W| max Indicates the extreme value of the weight. Represents the preset sequence c (b) The l-th value in the sequence, b represents the number of bits required to store the weight index, and the value of l is related to the preset sequence c. (b) It is related to the length L.

14. The apparatus according to any one of claims 10-13, wherein, The model processing unit is also used for: When performing model inference using the target model, new weight parameters that match the weight index are determined; the storage space occupied by the new weight parameters is greater than the storage space occupied by the weight index. Model inference is performed based on new weight parameters that match the weight index.

15. The apparatus according to claim 14, wherein, The storage unit is also used to store a preset index table; The model processing unit is specifically used to determine new weight parameters that match the weight index based on a stored preset index table, so as to convert the weight index into weight parameters required for model inference.

16. The apparatus according to claim 14, wherein, The model processing unit is specifically used for: Based on the new weight parameters that match the weight index and the quantized activation parameters in the target model, dequantization is performed to utilize the dequantized target model for model inference.

17. The apparatus according to claim 16, wherein, The storage unit is also used to store size factors; The model processing unit is also used to acquire the stored size factor; and to quantize the activation parameters in the target model based on the size factor to obtain the quantized activation parameters.

18. The apparatus according to claim 16, wherein, The model processing unit is specifically used for: The new weight parameters, which match the weight index, are fused with the quantized activation parameters to perform inverse quantization based on the fusion result.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images