Model quantification method and device, electronic equipment, chip, storage medium and computer program product

Abstract

The embodiment of the invention provides a model quantization method, a model quantization device, electronic equipment, a chip, a storage medium and a computer program product, and the method comprises the steps: for any to-be-quantized unit of a Transform module in a to-be-quantized model, carrying out the transformation of a first activation value corresponding to the to-be-quantized unit according to a first vector corresponding to the to-be-quantized unit, and obtaining a first activation value corresponding to the to-be-quantized unit; obtaining a second activation value; the first vector is used for enhancing the weight of an important vocabulary in the first activation value; according to the first information, calling a quantization model to obtain a quantized weight of the to-be-quantized unit; the first information comprises a second activation value and a first weight; the first weight is the weight of the to-be-quantized unit before quantization.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of large model technology, and specifically to a model quantization method, a model quantization device, an electronic device, a chip, a storage medium, and a computer program product. Background Art

[0002] With the rapid development of big model technology, big models can be applied to multiple business scenarios. For big data scenarios, multiple big models need to be deployed to achieve the goal, which requires higher costs to support. Big models also require more computing power to obtain results, so they require more computing time and cannot meet business needs that require high real-time performance. Related technologies mainly use model weight quantization technology to construct smaller big models, but these general quantization methods cannot fully consider the data structure characteristics under different business scenarios, which will cause the accuracy of the quantized big models to decrease. Summary of the invention

[0003] Embodiments of the present application provide a model quantization method, a model quantization device, an electronic device, a chip, a storage medium, and a computer program product.

[0004] The model quantization method provided in the embodiment of the present application includes:

[0005] For any unit to be quantized of the Transformer module in the model to be quantized, transform the first activation value corresponding to the unit to be quantized according to the first vector corresponding to the unit to be quantized to obtain a second activation value; the first vector is used to enhance the weight of important words in the first activation value;

[0006] According to the first information, the quantization model is called to obtain the weight of the unit to be quantized after quantization; the first information includes: a second activation value and a first weight; the first weight is the weight of the unit to be quantized before quantization.

[0007] The model quantization device provided in the embodiment of the present application includes:

[0008] A processing unit: for transforming a first activation value corresponding to any unit to be quantized of the Transformer module in the model to be quantized according to a first vector corresponding to the unit to be quantized, to obtain a second activation value; the first vector is used to enhance the weight of important words in the first activation value;

[0009] The processing unit is also used to call the quantization model according to the first information to obtain the weight of the unit to be quantized after quantization; the first information includes: the second activation value and the first weight; the first weight is the weight of the unit to be quantized before quantization.

[0010] The electronic device provided in the embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any model quantization method provided in the embodiment of the present application.

[0011] The chip provided in the embodiment of the present application includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes any model quantization method provided in the embodiment of the present application.

[0012] The storage medium provided in the embodiments of the present application is used to store a computer program, and the computer program enables a computer to execute any model quantization method provided in the embodiments of the present application.

[0013] The computer program product provided in the embodiments of the present application includes a computer program, and when the computer program is executed by a processor, it implements any model quantization method provided in the embodiments of the present application.

[0014] Through the model quantization method provided in the embodiment of the present application, the activation value of the input quantization model is adjusted through the first vector, so that the quantization model can pay attention to the structural characteristics of the user data during the quantization process, so that the quantized large model can retain more important information of the user data during calculation, thereby improving the prediction accuracy of the quantized large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A schematic diagram of the implementation process of the model quantization method provided in the embodiment of the present application;

[0017] Figure 2 A business flow chart of 5G terminal replacement provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of a 5G terminal replacement model provided in an embodiment of the present application;

[0019] Figure 4 A schematic diagram of the construction process of a 5G terminal replacement model provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of the structure of a model quantization device provided in an embodiment of the present application;

[0021] Figure 6 A schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0022] Figure 7 A schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be noted that in the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the embodiments of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0025] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0026] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0027] As the variety of mobile business products increases, the marketing methods and channels for these products and services are also increasing. Traditional marketing channels are mainly based on outbound calls, inbound calls, and business halls. In addition, the marketing volume of marketing channels such as SMS and TV is also increasing. In addition to market expansion, as the marketing goals of existing business products continue to increase, the performance requirements for marketing models are also increasing, thereby strengthening and consolidating the market share of existing markets.

[0028] Mining the population for 5G terminal replacement business is one of the important businesses in mobile marketing. The existing 5G terminal replacement business population mining models are mainly based on data-driven artificial neural network models. These models are mainly based on the user numerical data characteristics in the business scenario, and other types of data are preprocessed or encoded into numerical data. With the development of big model technology, the 5G terminal replacement business population mining model combined with the big model can fully extract the effective information of the text data in the marketing scenario, and at the same time, the association of different fields of the original user feature structured data is modeled, which can improve the hit rate of the original 5G terminal replacement business population mining in some business scenarios.

[0029] However, the 5G terminal replacement business population mining model based on large models also faces some technical problems in practical applications. First, the weight parameters of the large model require a large amount of GPU memory resources to run. In scenarios with a large marketing population, multiple large models will be deployed to achieve the goal, which requires high costs to support. In addition, large models require more computing power to obtain results, so more computing time is required. In business scenarios with high real-time requirements such as business halls, response delays will occur, which cannot meet business needs that require high real-time performance.

[0030] In the related technologies, the methods to solve the problem of excessive consumption of computing resources caused by large number of parameters and lack of real-time performance of large models are mainly based on model compression technologies such as model weight quantization, pruning and distillation. These methods mainly aim at parameter compression and reasoning acceleration of large models. On the premise of keeping the model effect meeting the requirements, a model with smaller parameters is constructed to replace the original model. Among these methods, the method based on model weight quantization has higher stability and more universal application. Although the general weight quantization method can construct a smaller large model, these general quantization methods cannot fully consider the data structure characteristics of the 5G terminal replacement business population mining scenario, which will lead to a slight decrease in the hit rate of the precision marketing model. Therefore, it is necessary to optimize the general quantization method to reduce the accuracy loss caused by the quantization compression of model parameters.

[0031] Figure 1 A schematic diagram of the implementation flow of the model quantization method provided in the embodiment of the present application, such as Figure 1 As shown, the embodiment of the present application provides a model quantization method, which includes the following steps:

[0032] Step 101: for any unit to be quantized of the Transformer module in the model to be quantized, transform the first activation value corresponding to the unit to be quantized according to the first vector corresponding to the unit to be quantized to obtain a second activation value; the first vector is used to enhance the weight of important words in the first activation value.

[0033] In the embodiment of the present application, the model to be quantified may be a large language model. Exemplarily, the model to be quantified may be a 5G terminal replacement model. The 5G terminal replacement scenario is mainly based on user feature data to predict the probability of users applying for 5G terminal replacement in the future, and then through marketing means such as active inbound calls and passive outbound calls, high-value users who are more likely to apply for 5G terminal replacement services are screened. Reference Figure 2 , Figure 2The business flow chart of 5G terminal replacement provided in the embodiment of the present application, the process of 5G terminal replacement business mainly includes three stages. The first stage is the preprocessing stage of the 5G terminal replacement model, which mainly includes the demand analysis and task strategy formulation of the 5G terminal replacement construction, and the user feature data required for the training model based on historical user data and the annotation of whether to purchase 5G terminals in the future. The second stage is the construction stage of the 5G terminal replacement model, which is mainly completed based on the training data formed in the first stage. The third stage is the deployment and actual verification based on the model built in the second stage. First, the model is encapsulated and loaded with services to form a prediction system, which is deployed in the corresponding online and offline systems. Next, each business hall and relevant departments will use it and feedback the indicator results. If the verification indicator meets the requirements, continue to use it and implement it in depth. Otherwise, rebuild the 5G terminal replacement model and optimize the structure of the model.

[0034] The following is an introduction to user characteristic data in the 5G terminal replacement scenario. Where N represents the number of user data x i Represents user feature data with n field features, which are specifically composed as follows: in Indicates the text description of the jth field. Indicates the value of the jth field. When entering the 5G terminal replacement model, you need to Convert it into a string, and then convert it into a string according to j. Arrange them to form a long string: Indicates whether the user will apply for 5G terminal replacement service in the future. i The i-th component of is 1, indicating that the user has applied for the 5G terminal replacement service, and the i-th component is 0, indicating that the user has applied for the 5G terminal replacement service. Refer to Table 1, which is a user feature field table. The user data feature vector x i The different components of correspond to the characteristic fields in Table 1:

[0035]

[0036] Table 1

[0037] The user's feature categories need to be divided into four categories. The first category of continuous features belongs to the user's dynamic features of the month, which have a certain randomness as they grow over time. The second category of features belongs to static features, which have a certain trend as they grow over time. The third category of continuous features belongs to traffic features, which mainly characterize the user's traffic behavior of the month. The fourth category is the category of discrete features, which are generally static features and have no trend information.

[0038] refer to Figure 3 , Figure 3The schematic diagram of the structure of the 5G terminal switching model provided in the embodiment of the present application is shown in FIG. 5G terminal switching model mainly includes a Token conversion module, an Embedding module, a Transformer module and a Head module. The Token conversion module mainly converts x i The words in the data are mapped into corresponding serial numbers, and the mapping table is determined by the large model itself. The Embedding module mainly transforms the token sequence of user data into vectors to form an Embedding matrix representation. The Transformer module performs feature transformation on the Embedding matrix representation of user feature data. The Head module takes the features extracted by the Transformer module as input and outputs the confidence of the user in applying for the 5G terminal replacement service. The Head module is constructed based on the 5G terminal replacement service scenario.

[0039] Specifically, the Token conversion module: input user feature data x i , output x i The token conversion layer mainly converts x i Split into words and then map to corresponding tokens. In the 5G terminal replacement model, the field value Independently constitute a word. Embedding module: user data x i The token sequence is used as input and the user feature matrix is ​​output Where b represents the number of tokens in the token sequence. i In the token sequence in , the jth token corresponds to the feature matrix The jth row vector in . The Transformer module is mainly composed of L Transformer submodules, each of which has the same network structure. The input of the lth Transformer module is recorded as Output is recorded as The Transformer submodule mainly includes the attention module (denoted as ATT l ) and FFN module (denoted as FFN l ). Among them, the weight parameters of the attention module include the query weight Key Weight Sum weight Input notation Output is recorded as The calculation process of the ATT module is: Among them, Q l is the query matrix, K l is the bond matrix, V lis a value matrix. According to the parameter n_heads of the attention head number of the large model, Q l , K l 、V l According to the column dimension, it is evenly divided into n_heads sub-matrices, denoted as Q lj , K lj 、V lj , where j = 1…n_heads. For each j, calculate the following formula: Where d∈R, M is a lower triangular matrix, the lower triangular part is 0 (including the diagonal), and the upper triangular part is -∞. After the calculation is completed, the calculation result is Concatenate according to the column dimension to form the matrix X'. Perform matrix multiplication calculations, the output of the ATT module The input of the FFN module is the output of the ATT module The output of the FFN module is The FFN module mainly includes four weight parameters, namely the weight matrix W of the first fully connected layer l 1 ∈R e×t , the weight matrix W of the second fully connected layer l 2 ∈R t×e , the bias vector of the first fully connected layer The bias vector of the second fully connected layer The calculation formula of the FFN module is as follows: Where f represents the relu activation function.

[0040] Head layer (denoted as H(x)): The input is the output of the Transformer module The output is the confidence of 5G terminal replacement. The calculation formula of the Head layer is as follows: Among them, W H and b H are the weight matrix and bias vector of the Head layer respectively, Represents the matrix A one-dimensional vector concatenated by row vectors.

[0041] The model quantization method provided in the embodiment of the present application can be used for weight W l 1 , W l 2 Quantization is performed, and the unit to be quantized can be a weight W l 1 , W l 2 The unit where you are located.

[0042] In the embodiment of the present application, the quantization model may adopt the GPTQ (Generative Pre-trained Transformer-Q) model. GPTQ is a quantization technology for generating pre-trained Transformer models. The process of the GPTQ quantization method mainly includes the following steps: designing transformations f1 and f, and performing normalization transformations on the first activation value X and the weight to be quantized W respectively: Will and As input, it is calculated by the GPTQ model and output in Each element in is of int4 integer type. The construction is based on the following goals:

[0043] In the embodiment of the present application, the input X of the GPTQ model is optimized to make the quantization error of the GPTQ smaller. In the embodiment of the present application, the first activation value X of the quantized unit is transformed by the first vector M to obtain the second activation value X o , the first vector is used to enhance the weight of important words in the first activation value. Exemplarily, the transformation is performed using the following calculation formula: X0=M·X, where each element in M ​​is multiplied by the corresponding row vector of X.

[0044] For example, if the weight Quantized, the first activation value is If the weight W l 1 Quantized, the first activation value is

[0045] In the embodiment of the present application, the model to be quantified includes a Token conversion module and an Embedding module. i ), the output of the Embedding module E(x i ) and the output T of the transformer submodule l (x) constructing a first vector for the gradient of the row vector in the first activation value to enhance the weight of the important words in the first activation value. The output of the Token conversion module, the Embedding module, the output of the transformer submodule, and the first activation value can be input into the user feature data importance model to construct the first vector.

[0046] The following describes how to construct the first vector:

[0047] First, select the key tokens with key information for the business scenario so that the row vector corresponding to the quantization error objective function in the input X has a larger weight. It should be noted that in any submodule of the Transformer module, the row vector of the matrix multiplication input X has a one-to-one correspondence with the token sequence output by the Token conversion module.

[0048] Construct the second vector M1, which is the Token importance vector, used to enhance the weight of the content word; the i-th component of M1 corresponds to Token(x i ) in M1. If the token is a content word, the i-th component of M1 is 1, otherwise it is 0; illustratively, content words include: words that are helpful for explaining the field name and the corresponding values ​​of the field. Take the field "Shopping APP Traffic" in Table 1 as an example, where "shopping", "APP", and "traffic" are content words, and the corresponding components in M1 are set to 1; "category" is a function word, and the corresponding component in M1 is set to 0; the value corresponding to the field "Shopping APP Traffic" is a content word, and the corresponding component in M1 is set to 1.

[0049] Secondly, the weights of words that have high semantic similarity with other words or appear frequently are attenuated to reduce the deviation of the objective function caused by the uneven number of different words.

[0050] Through the output E(x i ), construct the third vector M2 to reduce the repetition rate of non-core words. Construct vector R = E(x i )*E(x i ) T , construct a vector R1, where the jth component of R1 That is, the absolute values ​​of the elements of the matrix R are summed row by row, where a ij Indicates the weights between different tokens of user feature data, a ij Defined based on the type of user features or the semantics of the text. For example, a ij The weight definition criteria are as follows:

[0051] When the i-th and j-th tokens represent non-core words, a ij Defined as 1.

[0052] When one of the i-th and j-th tokens is a non-core word and the other is a core word, a ij Defined as 0.5.

[0053] When the i-th and j-th tokens are both core words, a ij Defined as 0.25.

[0054] When the i-th, j-th token has a numerical feature representing user feature data, a ij Defined as 0.

[0055] Here, core vocabulary refers to the vocabulary that identifies the characteristics of the field. ij The weight definition criteria are mainly based on the fact that core words can be repeated more often, and the numerical features themselves do not have the problem of semantic repetition, and the importance is high, so there is no need to remove the repetition. In actual application, other a ij The weight definition criteria are not limited in the present application. For example, taking the field "shopping APP traffic" as an example, "APP" and "traffic" are core words, and "shopping" and "category" are non-core words.

[0056] That is, the elements of vector R1 are normalized, and the hyperparameter w∈(0,1).

[0057] It should be noted that the row vectors of the input matrices of all Transformer submodules have a one-to-one correspondence with the Token sequence in order, so M2 is shared by all Transformer submodules.

[0058] Finally, the output T of the lth transformer submodule l (x) has different sensitivities to the row vectors in X, and we need to select the best l (x) is the row vector of X that has a greater impact. Therefore, a fourth vector M3 is constructed to measure its sensitivity. The fourth vector is used to enhance the weight of the words that are highly sensitive to the output of the unit to be quantized. The calculation method of M3 is as follows: calculate That is, the absolute values ​​of the elements of matrix G are summed row by row.

[0059] Combining M1, M2 and M3, the output of the user feature data importance model is defined as M = M1·M2·M3.

[0060] It should be noted that, in actual application, the first vector M can also be constructed by one or more vectors of M1, M2 and M3. For example, M can be constructed only by M1 and M2, or M can be constructed only by M2. This embodiment does not limit this.

[0061] Based on this, in an optional implementation manner of the present application, the model to be quantified includes: a Token conversion module and an Embedding module; the method further includes:

[0062] Constructing a second vector according to the output of the Token conversion module; the second vector is used to enhance the weight of the content word;

[0063] Constructing a third vector according to the output of the Embedding module; the third vector is used to reduce the repetition rate of non-core words;

[0064] Constructing a fourth vector according to the output of the unit to be quantized and the first activation value; the fourth vector is used to enhance the weight of words that are highly sensitive to the output of the unit to be quantized;

[0065] The first vector is constructed according to the second vector, the third vector and the fourth vector.

[0066] Step 102: According to the first information, a quantization model is called to obtain the weight of the unit to be quantized after quantization; the first information includes: a second activation value and a first weight; the first weight is the weight of the unit to be quantized before quantization.

[0067] In an embodiment of the present application, a quantization model is called according to the second activation value and the first weight to obtain the quantized weight of the unit to be quantized, and the second activation value is an activation value of the weight of important vocabulary that has been enhanced, so that the final quantized model can pay more attention to the important information of text and numerical values, thereby improving the accuracy of the quantization model.

[0068] In the embodiment of the present application, the distribution of user feature data and weights is often incompatible with the quantization model, which will lead to the problem of large quantization error. Before calling the quantization model, the second activation value and the first weight can be further processed to adjust the distribution of the second activation value and the first weight, thereby improving the compatibility of the activation value with the quantization model and improving the prediction accuracy of the quantization model.

[0069] Based on this, in an optional implementation manner of the present application, the first information further includes: a fifth vector, and before calling the quantization model, further includes:

[0070] According to a fifth vector, the second activation value and the first weight are transformed to obtain a third activation value and a second weight; the fifth vector is used to adjust the distribution of the second activation value and the first weight to improve the compatibility of the second activation value and the first weight with the quantization model.

[0071] The following describes how the fifth vector is constructed:

[0072] Through the distribution optimization model, the second activation value X0 and the first weight W are used as inputs, and the fifth vector S is output, so that the distribution of X0 and W has a higher compatibility with the GPTQ model, thereby reducing the quantization error of GPTQ. To achieve this goal, the distribution optimization model considers the distribution of X0 and the distribution of W respectively, constructs the sixth vector S1 and the seventh vector S2 respectively, and finally merges S1 and S2 to generate S.

[0073] The construction method of S1 is as follows. In the calculation phase of the GPTQ model, the correlation matrix needs to be calculated and its inverse. In the 5G terminal replacement model, due to the relatively small amount of semantic data, redundancy may occur between different components (i.e., column vectors) of the activation value X of the trained model, so that some components can be approximately represented by other components. This phenomenon will cause the rank of the correlation matrix R2 to be relatively low, and the size of the non-zero eigenvalues ​​of the matrix will differ significantly. Therefore, according to the original GPTQ calculation method, adding αI (I is the unit matrix) for regularization (R2 is replaced by R2+αI) to calculate the inverse of R2 will lead to instability. This regularization cannot correct the calculation deviation caused by the large difference in the non-zero eigenvalues ​​of R2.

[0074] In the embodiment of the present application, the row vectors in X0 are first clustered by the semantic information of the corresponding tokens to avoid the large difference in singular values ​​due to the high similarity of the Embedding vector representations of the corresponding tokens of words with similar semantic information. For example, referring to the user data features in Table 1, the clustering steps are as follows:

[0075] First, the row vectors corresponding to the same token are directly averaged to obtain a new row vector. Second, define the row vector x i and x j The distance between them is as follows: t ij ||x i -x j ||, where the row vector x i and x j When the corresponding tokens belong to the description statements of the same type of features, t ij is 1, otherwise, t ij is 10.

[0076] Consider the set of all row vectors in X0. Each time, take the average of the two row vectors with the smallest distance, put the average into the set, and then remove these two row vectors until the distance of each row vector is greater than the given threshold. At this time, the remaining row vectors in this set form a new matrix based on Generate a new correlation matrix

[0077] In the embodiment of the present application, vector S1 is constructed according to the new correlation matrix, so that The non-zero eigenvalues ​​of are as close as possible. S1 is optimized by the following objective function: Among them, x1, x2 are The eigenvectors corresponding to the maximum eigenvalue and the minimum non-zero eigenvalue are obtained by iteratively solving the minimum value problems of x1, x2 and S1. Perform orthogonal decomposition to calculate the solution for a given S1.

[0078] The construction method of S2 is as follows: regularize the row vectors of W to reduce the differences in the lengths of the row vectors of W, thereby reducing the quantization error. By calculating the maximum absolute value of the components of W row by row, we get the vector S2 can be obtained by the following formula: a>0, b>0, a and b are hyper parameters.

[0079] The distribution optimization model outputs the final result S=S1*S2.

[0080] Based on this, in an optional implementation manner of the present application, the method further includes: performing a clustering operation on the second activation value according to token semantic information corresponding to a row vector in the second activation value to obtain a fourth activation value;

[0081] Constructing a sixth vector according to a correlation matrix; wherein the correlation matrix is ​​obtained based on the fourth activation value;

[0082] Regularizing the row vector of the first weight to obtain a seventh vector;

[0083] The fifth vector is constructed according to the sixth vector and the seventh vector.

[0084] In the embodiment of the present application, the second activation value X0 and the first weight W are transformed according to the fifth vector S to obtain a third activation value X1=X0·S, and a second weight W1=S -1 ·W.

[0085] In the embodiment of the present application, the quantization model is called according to X1 and W1 to obtain the quantized weights, so that the input distribution of the quantization model has higher compatibility with the quantization model, thereby improving the quantization accuracy.

[0086] In an embodiment of the present application, the GPTQ model adopts a greedy strategy and quantizes in sequence according to the row vector order of the weights. According to the algorithm principle of the GPTQ model, the quantization error caused by the row vectors that are quantized first is smaller. Therefore, in an embodiment of the present application, the row vector of the weights input to the GPTQ model is adjusted through the first matrix, so that the row vectors that have a greater impact on the output are quantized first. Correspondingly, the column vector of the activation value is adjusted synchronously according to the adjustment method of the row vector of the weights.

[0087] Based on this, in an optional implementation manner of the present application, the first information further includes: a first matrix, and before calling the quantization model, further includes:

[0088] Calculating a normalized parameter vector offset and a scaling factor for the second weight;

[0089] The second weight, the offset and the scaling factor are used to obtain a third weight;

[0090] The third activation value and the third weight are transformed by a first matrix to obtain a fifth activation value and a fourth weight; the first matrix is ​​used to sort the column vectors of the third activation value and the row vectors of the third weight in descending order based on the weights of the influence of each row vector of the third weight on the output of the quantization unit.

[0091] In the embodiment of the present application, the quantization range is determined, the normalized parameter vector offset Offset and the scaling factor Scale of the second weight W1 are calculated, and the third weight W2 = (W1-Offset) / Scale is further obtained. The operations of Offset and Scale on W1 are both column-by-column operations, and each column of W1 is operated with the corresponding components of Offset and Scale), and each element in W2 is within the range of [-8, 7]. It should be noted that the range of can be designed according to actual conditions. For example, it can also be set to the range of [0, 15]. The embodiment of the present application does not limit this.

[0092] In the embodiment of the present application, the weight component importance model takes X1 and W2 as input to obtain a first matrix P. The row vectors of W2 and the column vectors of X1 are sorted by P, so that the row vectors with a greater impact on the model output are placed in front of W2. GPTQ can give priority to quantizing more important row vectors, thereby obtaining an overall smaller quantization error.

[0093] Since the gradient is an important indicator for measuring the sensitivity of the model output, the weight component importance model processing flow is as follows:

[0094] First, calculate the gradient of the output of the Transformer submodule with respect to W2 and get the gradient matrix

[0095] Secondly, calculate W g The maximum absolute value of the row vectors gives vector T0.

[0096] Finally, the row vectors of W2 are arranged according to the size of T0, so that the row vectors of W2 have larger T0 component values ​​corresponding to the row vectors from top to bottom. This arrangement matrix is ​​the output P.

[0097] Based on this, in an optional implementation manner of the present application, the method further includes:

[0098] Calculating the maximum absolute value of a row vector of a first gradient matrix to obtain an eighth vector; wherein the first gradient matrix is ​​a gradient of the unit to be quantized output with respect to the third weight;

[0099] According to the size of the eighth vector, the row vectors of the third weight are arranged in descending order to obtain the first matrix.

[0100] In the embodiment of the present application, the row vector of W2 is adjusted by the following calculation method to obtain the fourth weight W3=P*W2. After adjusting the row vector of the weight, the column vector of the third activation value also needs to be adjusted accordingly to obtain the fifth activation value X2=X1*P T , input X2 and W3 into the quantization model to obtain the fifth weight W int4 , the order of the row vectors in the fifth weight is different from the row vectors of the weights of the model to be quantized. Therefore, it is necessary to inversely transform the fifth weight through the first matrix to obtain the final quantized weight W f =P T *W int4 .

[0101] Based on this, in an optional implementation manner of the present application, calling the quantization model to obtain the quantized weight of the unit to be quantized includes:

[0102] Inputting the fifth activation value and the fourth weight into the quantization model to obtain a fifth weight;

[0103] The fifth weight is inversely transformed through the first matrix to obtain the quantized weight.

[0104] Furthermore, the parameter weight W f , Offset and Scale are deployed to the model to be quantized for post-quantization reasoning.

[0105] refer to Figure 4 , Figure 4 A schematic diagram of the construction process of the 5G terminal replacement model provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the construction process of the 5G terminal replacement model is mainly divided into two parts. The first part is the construction of a high-precision training model, which is mainly based on the existing neural network model for training. The second part is model compression. The original GPTQ model compression process directly normalizes the activation values ​​and weights corresponding to the data in the neural network, and then calls the traditional GPTQ method to convert the weights from high precision to int4 low precision. The model quantization method adopted in the embodiment of the present application optimizes and improves the quantization accuracy through three ways: enhancing the importance of important vocabulary through M, optimizing data distribution through S, and adjusting the weight row vector through P, thereby improving the inference accuracy of the quantized model.

[0106] Refer to Table 2, which is a comparison table between the prediction accuracy of the model quantized by the model quantization method of the present application, the accuracy of the model quantized by the original GPTQ method, and the accuracy of the quantized model.

[0107]

[0108] Table 2

[0109] As shown in Table 2, the model quantization method proposed in the embodiment of the present application can reduce the model memory consumption while ensuring that the model hit rate remains unchanged. In the model used in the embodiment of the present application, the video memory can be reduced from 68G to 26G, and the resources of 3 V100 can be reduced to the resources of 1 V100.

[0110] The model quantization method provided in the embodiment of the present application enhances the weight of important words in the activation value by setting the first vector, so that when performing operations based on the model weights after GPTQ quantization, important information of text and numerical values ​​is retained as much as possible; by setting the fifth vector, the distribution of the second activation value and the first weight is adjusted to improve the compatibility of the second activation value and the first weight with the quantization model, providing GPTQ with a stable data and weight distribution, and improving the quantization accuracy of GPTQ; through the first matrix, based on the weight of each row vector of the third weight affecting the output of the quantization unit, the column vector of the third activation value and the row vector of the third weight are sorted in descending order, so that GPTQ can compensate for the important parts of the weight more fully, thereby improving the quantization accuracy of GPTQ.

[0111] The present application also provides a model quantization device, referring to Figure 5 , Figure 5 This is a schematic diagram of the structure of a model quantization device provided in an embodiment of the present application. The model quantization device in this embodiment includes:

[0112] A processing unit: for transforming a first activation value corresponding to any unit to be quantized of the Transformer module in the model to be quantized according to a first vector corresponding to the unit to be quantized, to obtain a second activation value; the first vector is used to enhance the weight of important words in the first activation value;

[0113] The processing unit is also used to call the quantization model according to the first information to obtain the weight of the unit to be quantized after quantization; the first information includes: the second activation value and the first weight; the first weight is the weight of the unit to be quantized before quantization.

[0114] In an embodiment of the present application, the model to be quantified includes: a Token conversion module and an Embedding module; the processing unit is used to construct a second vector according to the output of the Token conversion module; the second vector is used to enhance the weight of the content words; a third vector is constructed according to the output of the Embedding module; the third vector is used to reduce the repetition rate of non-core vocabulary; a fourth vector is constructed according to the output of the unit to be quantified and the first activation value; the fourth vector is used to enhance the weight of vocabulary that is highly sensitive to the output of the unit to be quantified; the first vector is constructed according to the second vector, the third vector and the fourth vector.

[0115] In an embodiment of the present application, the first information also includes: a fifth vector. Before calling the quantization model, the processing unit is used to transform the second activation value and the first weight according to the fifth vector to obtain a third activation value and a second weight; the fifth vector is used to adjust the distribution of the second activation value and the first weight to improve the compatibility of the second activation value and the first weight with the quantization model.

[0116] In an embodiment of the present application, the processing unit is used to: perform a clustering operation on the second activation value according to the token semantic information corresponding to the row vector in the second activation value to obtain a fourth activation value; construct a sixth vector according to a correlation matrix; wherein the correlation matrix is ​​obtained based on the fourth activation value; regularize the row vector of the first weight to obtain a seventh vector; and construct the fifth vector according to the sixth vector and the seventh vector.

[0117] In an embodiment of the present application, the first information also includes: a first matrix, before calling the quantization model, the processing unit: is used to calculate the normalized parameter vector offset and scaling factor of the second weight; the second weight, offset and scaling factor, to obtain the third weight; the third activation value and the third weight are transformed by the first matrix to obtain the fifth activation value and the fourth weight; the first matrix is ​​used to sort the column vector of the third activation value and the row vector of the third weight in descending order based on the weight of the influence of each row vector of the third weight on the output of the quantization unit.

[0118] In an embodiment of the present application, the processing unit is used to calculate the maximum absolute value of the row vector of the first gradient matrix to obtain the eighth vector; the first gradient matrix is ​​the gradient of the unit to be quantized output with respect to the third weight; according to the size of the eighth vector, the row vector of the third weight is arranged in descending order to obtain the first matrix.

[0119] In an embodiment of the present application, the processing unit is used to input the fifth activation value and the fourth weight into the quantization model to obtain the fifth weight; and perform an inverse transformation on the fifth weight through the first matrix to obtain the quantized weight.

[0120] Those skilled in the art should understand that Figure 5 The implementation functions of each unit in the model quantization device shown can be understood by referring to the relevant description of the aforementioned method. Figure 5 The functions of each unit in the model quantization device shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0121] Figure 6 It is a schematic structural diagram of an electronic device 600 provided in an embodiment of the present application. Figure 6 The electronic device 600 shown includes a processor 610, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0122] Alternatively, if Figure 6 As shown, the electronic device 600 may further include a memory 620. The processor 610 may call and run a computer program from the memory 620 to implement the method in the embodiment of the present application.

[0123] The memory 620 may be a separate device independent of the processor 610 , or may be integrated into the processor 610 .

[0124] Alternatively, if Figure 6 As shown, the electronic device 600 may further include a transceiver 630, and the processor 610 may control the transceiver 630 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0125] The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include an antenna, and the number of the antennas may be one or more.

[0126] The electronic device 600 may specifically be a model quantization device in an embodiment of the present application, and the electronic device 600 may implement the corresponding processes implemented by the model quantization device in each method in the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0127] Exemplarily, an embodiment of the present application further provides a computer program product, including a computer program, and the computer program can be executed by the processor 610 of the communication device 600 to complete the steps of any of the aforementioned methods.

[0128] Figure 7It is a schematic structural diagram of the chip of an embodiment of the present application. Figure 7 The chip 700 shown includes a processor 710, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0129] Alternatively, if Figure 7 As shown, the chip 700 may further include a memory 720. The processor 710 may call and run a computer program from the memory 720 to implement the method in the embodiment of the present application.

[0130] The memory 720 may be a separate device independent of the processor 710 , or may be integrated into the processor 710 .

[0131] Optionally, the chip 700 may further include an input interface 730. The processor 710 may control the input interface 730 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0132] Optionally, the chip 700 may further include an output interface 740. The processor 710 may control the output interface 740 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0133] The chip can be applied to the electronic device 600 in the embodiment of the present application, and the chip can implement the corresponding processes implemented by the electronic device 600 in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0134] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0135] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software units in the decoding processor are combined and executed. The software unit can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0136] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0137] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0138] The present application also provides a storage medium for storing a computer program. The storage medium can be applied to the electronic device 600 in the present application, and the computer program enables the computer to execute the corresponding process implemented by the electronic device 600 in each method of the present application, which will not be described in detail for the sake of brevity.

[0139] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0144] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or an electronic device 600, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model quantization method, characterized in that: include: For any unit to be quantized of the Transformer module in the model to be quantized, transform the first activation value corresponding to the unit to be quantized according to the first vector corresponding to the unit to be quantized to obtain a second activation value; The first vector is used to enhance the weight of important words in the first activation value; According to the first information, a quantization model is called to obtain a quantized weight of the unit to be quantized; The first information includes: a second activation value and a first weight; The first weight is the weight of the unit to be quantized before quantization.

2. The method according to claim 1, characterized in that The model to be quantified includes: a Token conversion module and an Embedding module; the method also includes: Constructing a second vector according to the output of the Token conversion module; the second vector is used to enhance the weight of the content word; Constructing a third vector according to the output of the Embedding module; the third vector is used to reduce the repetition rate of non-core words; Constructing a fourth vector according to the output of the unit to be quantized and the first activation value; the fourth vector is used to enhance the weight of words that are highly sensitive to the output of the unit to be quantized; The first vector is constructed according to the second vector, the third vector and the fourth vector.

3. The method according to claim 1, characterized in that The first information also includes: a fifth vector, and before calling the quantization model, also includes: According to a fifth vector, the second activation value and the first weight are transformed to obtain a third activation value and a second weight; the fifth vector is used to adjust the distribution of the second activation value and the first weight to improve the compatibility of the second activation value and the first weight with the quantization model.

4. The method according to claim 3, characterized in that Also includes: Performing a clustering operation on the second activation value according to token semantic information corresponding to the row vector in the second activation value to obtain a fourth activation value; Constructing a sixth vector according to a correlation matrix; wherein the correlation matrix is ​​obtained based on the fourth activation value; Regularizing the row vector of the first weight to obtain a seventh vector; The fifth vector is constructed according to the sixth vector and the seventh vector.

5. The method according to claim 3, characterized in that: The first information also includes: a first matrix, and before calling the quantization model, also includes: Calculating a normalized parameter vector offset and a scaling factor for the second weight; The second weight, the offset and the scaling factor are used to obtain a third weight; The third activation value and the third weight are transformed by a first matrix to obtain a fifth activation value and a fourth weight; the first matrix is ​​used to sort the column vectors of the third activation value and the row vectors of the third weight in descending order based on the weights of the influence of each row vector of the third weight on the output of the quantization unit.

6. The method according to claim 5, characterized in that Also includes: Calculating the maximum absolute value of a row vector of a first gradient matrix to obtain an eighth vector; wherein the first gradient matrix is ​​a gradient of the unit to be quantized output with respect to the third weight; According to the size of the eighth vector, the row vectors of the third weight are arranged in descending order to obtain the first matrix.

7. The method according to claim 5, characterized in that The calling of the quantization model to obtain the quantized weight of the unit to be quantized includes: Inputting the fifth activation value and the fourth weight into the quantization model to obtain a fifth weight; The fifth weight is inversely transformed through the first matrix to obtain the quantized weight.

8. A model quantization device, characterized in that: include: A processing unit: for transforming a first activation value corresponding to any unit to be quantized of the Transformer module in the model to be quantized according to a first vector corresponding to the unit to be quantized, to obtain a second activation value; The first vector is used to enhance the weight of important words in the first activation value; The processing unit is further configured to call a quantization model according to the first information to obtain a quantized weight of the unit to be quantized; The first information includes: a second activation value and a first weight; The first weight is the weight of the unit to be quantized before quantization.

9. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the model quantization method as described in any one of claims 1 to 7.

10. A chip, characterized in that: include: A processor, used to call and run a computer program from a memory, so that a device equipped with the chip executes the model quantization method described in any one of claims 1 to 7.

11. A storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the model quantization method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the model quantization method according to any one of claims 1 to 7.

Images