Data processing method and device

Abstract

The invention discloses a data processing method and device, and the method comprises the steps: obtaining a first data set of a target model, the first data set comprises at least one group of initial weight data, each group of initial weight data corresponds to a network layer of the target model, and the target model is a large language model used for processing an inference task; a first binarization order corresponding to the initial weight data is determined according to first importance information of the initial weight data, the first importance information represents sensitivity of the corresponding initial weight data to output probability distribution of the target model, and the output probability distribution is used for determining a processing result of the target model; and performing binarization processing on the corresponding initial weight data according to the first binarization order to obtain first quantized data corresponding to the initial weight data, and processing the reasoning task based on the first quantized data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and apparatus. Background Technology

[0002] Large language models are now widely used to handle reasoning tasks in various scenarios. However, large language models typically contain a large amount of weight data. When a device calls a large language model to process a reasoning task, it needs to use this weight data to calculate the input data (such as images, text, etc.) corresponding to the reasoning task. The large amount of weight data causes large language models to consume a significant amount of storage resources on the device.

[0003] Some related techniques can reduce the precision of weight data in large language models by reducing the storage space occupied by each weight data, thereby reducing the storage resources occupied by the large language model. This processing technique is called model quantization of large language models.

[0004] However, the accuracy of large language models after model quantization will decrease when handling reasoning tasks. Summary of the Invention

[0005] Therefore, this application discloses the following technical solution:

[0006] The first aspect of this application provides a data processing method, including:

[0007] A first dataset of the target model is obtained, the first dataset including at least one set of initial weight data, each set of the initial weight data corresponding to a network layer of the target model, the target model being a large language model for processing inference tasks;

[0008] The first binarization order corresponding to the initial weight data is determined based on the first importance information of the initial weight data. The first importance information characterizes the sensitivity of the corresponding initial weight data to the output probability distribution of the target model. The output probability distribution is used to determine the processing result of the target model.

[0009] The initial weight data is binarized according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, so as to process the inference task based on the first quantized data.

[0010] Optionally, the first importance information includes the covariance of the log-likelihood gradient, which is the gradient of the logarithm of the output probability distribution relative to the initial weight data.

[0011] Optionally, the step of binarizing the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data includes:

[0012] The average value of the initial weight data set is determined as the quantization mean;

[0013] Based on the initial weight data and the quantization mean, determine the binarization matrix and the quantization scaling factor;

[0014] If the number of binarized matrices obtained is less than the first binarization order, a new quantization mean, binarization matrix, and quantization scaling factor are determined based on the deviation of the estimated weight data relative to a set of initial weight data, until a number of binarized matrices corresponding to the first binarization order are obtained; the estimated weight data is determined based on the obtained quantization mean, binarization matrix, and quantization scaling factor.

[0015] The first quantization data includes at least one quantization mean, at least one quantization scaling factor, and at least one binarization matrix.

[0016] Optionally, the target model includes multiple sub-models with different processing capabilities;

[0017] The first data set for obtaining the target model includes:

[0018] Obtain the weight data for multiple sub-models;

[0019] The weight data of the multiple sub-models are fused and processed to obtain a first data set, wherein the weight data contained in the first data set reflects the common data processing capabilities of the multiple sub-models;

[0020] Wherein, each set of initial weight data in the first data set is obtained by fusing data from the corresponding network layers contained in the multiple sub-models;

[0021] The deviation between the weight data of the sub-model and the weight data contained in the first dataset is used as the residual weight data of the sub-model, and the residual weight data reflects the unique data processing capability of the corresponding sub-model.

[0022] Optionally, the fusion processing of the weight data of the multiple sub-models to obtain a first data set includes:

[0023] Determine a second importance information for each of the sub-models, the second importance information characterizing the sensitivity of the weight data of the sub-model to the output probability distribution of the target model;

[0024] The weight data of multiple sub-models are fused based on the second importance information to obtain a first data set.

[0025] Optional, also includes:

[0026] The residual weight data of each sub-model is binarized to obtain the second quantized data corresponding to each sub-model;

[0027] The method for binarizing the residual weight data is different from the method for binarizing the initial weight data.

[0028] Optionally, obtaining the first data set of the target model includes:

[0029] Identify at least one target network layer among multiple network layers of the target model;

[0030] The initial weight data corresponding to the target network layer is obtained as the first data set of the target model.

[0031] Optionally, the step of binarizing the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data includes:

[0032] The weighted smoothing coefficients are determined based on the calibration data used for binarization processing;

[0033] The initial weight data is scaled according to the weight smoothing coefficient to obtain scaled weight data;

[0034] The scaling weight data is binarized according to the first binarization order corresponding to the initial weight data to obtain the first quantized data corresponding to the initial weight data.

[0035] Optionally, the step of binarizing the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data includes:

[0036] For each set of initial weight data, the weight data of different parts of the set of initial weight data are binarized according to the third binarization order and the first binarization order corresponding to the set of initial weight data to obtain the first quantized data corresponding to the set of initial weight data.

[0037] The first binarization order and the third binarization order are different, and the influence of different parts of the weight data in the initial set of weight data on the processing result of the target model is different.

[0038] A second aspect of this application provides a data processing apparatus, comprising:

[0039] The obtaining unit is used to obtain a first data set of the target model. The first data set includes at least one set of initial weight data. Each set of initial weight data corresponds to a network layer of the target model. The target model is a large language model for processing inference tasks.

[0040] The determining unit is configured to determine the first binarization order corresponding to the initial weight data based on the first importance information of the initial weight data, wherein the first importance information characterizes the sensitivity of the corresponding initial weight data to the output probability distribution of the target model, and the output probability distribution is used to determine the processing result of the target model;

[0041] The processing unit is configured to perform binarization processing on the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, so as to process the inference task based on the first quantized data. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart of a method for binarizing the initial weight data based on the first binarization order provided in this application embodiment;

[0045] Figure 3 This is a schematic diagram of an embodiment of the present application providing a reasoning task based on a first quantized data processing task;

[0046] Figure 4 This is a schematic diagram illustrating the fusion of weight data from multiple sub-models, provided in an embodiment of this application.

[0047] Figure 5 This is a flowchart illustrating a method for fusing weight data from multiple sub-models, as provided in an embodiment of this application.

[0048] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] This embodiment provides a data processing method; please refer to [link / reference]. Figure 1 Here is a flowchart of the method, which may include the following steps.

[0051] S101, Obtain the first data set of the target model. The first data set includes at least one set of initial weight data. Each set of initial weight data corresponds to a network layer of the target model. The target model is a large language model used to process inference tasks.

[0052] The target model can be any data processing model built and trained based on deep learning techniques. For example, the target model can be a Large Language Model (LLM).

[0053] The target model can contain multiple network layers. Each network layer can include one or more matrices composed of weight data, denoted as weight matrices. In this embodiment, the weight data in all weight matrices contained in a network layer can be collectively referred to as a set of initial weight data corresponding to that network layer.

[0054] For example, network layer 1 includes a weight matrix, and all the weight data contained in this weight matrix are collectively referred to as the initial weight data corresponding to network layer 1.

[0055] The method in this embodiment can binarize all or part of the network layers in the target model. The first data set may include the initial weight data corresponding to the network layers in the target model that need to be binarized. For example, if the target model consists of 20 network layers, the first data set may contain 20 sets of initial weight data corresponding to these 20 network layers, or it may only contain 10 sets of initial weight data corresponding to 10 network layers, while the initial weight data of the other 10 network layers may not be binarized.

[0056] The initial weight data can be weight data represented in a high-precision data format, for example, the initial weight data can be weight data in 32-bit floating-point (float) format.

[0057] S102, determine the first binarization order corresponding to the initial weight data based on the first importance information of the initial weight data. The first importance information characterizes the sensitivity of the corresponding initial weight data to the output probability distribution of the target model. The output probability distribution is used to determine the processing result of the target model.

[0058] First importance information can be a parameter characterizing the sensitivity of the corresponding initial weight data to the output probability distribution of the target model. For any initial weight data, the larger the value of its first importance information, the greater the sensitivity of the initial weight data to the output probability distribution of the target model, or in other words, the greater the influence of the initial weight data on the output probability distribution of the target model; the smaller the value of its first importance information, the smaller the sensitivity of the initial weight data to the output probability distribution of the target model, or in other words, the smaller the influence of the initial weight data on the output probability distribution of the target model.

[0059] The relationship between the output probability distribution and the processing result is as follows: After processing the input data, the target model obtains an output probability distribution, which consists of the probability values ​​of multiple candidate processing results. Based on the probability values ​​in the output probability distribution, one of the multiple candidate processing results can be determined as the processing result of the target model. For example, a large language model can infer an output probability distribution containing the probability values ​​of multiple candidate character codes based on the input character codes, and then select the character code with the highest probability value as the processing result output of the large language model.

[0060] In step S102, the overall importance information of a network layer can be obtained by fusing the first importance information of a set of initial weight data corresponding to that network layer, on a layer-by-layer basis. For example, the average, median, or other indicators of the first importance information of a set of initial weight data corresponding to that network layer can be calculated, and the calculation result can be used as the overall importance information of that network layer. Then, based on the overall importance information corresponding to the network layer, the first binarization order used for a set of initial weight data corresponding to that network layer is determined. The first binarization order corresponding to different sets of initial weight data can be the same or different.

[0061] For any network layer, the first binarization order of that network layer is positively correlated with its overall importance information. A network layer with higher overall importance information corresponds to a higher first binarization order, and vice versa.

[0062] The first binarization order can be an integer greater than or equal to 1.

[0063] This embodiment does not limit the method for determining the first binarization order of each network layer based on the overall importance information. As some examples, an interval for the first binarization order can be set, for example, [1, 5]. Based on the aforementioned positive correlation condition, the overall importance information of each network layer can be mapped to different first binarization orders within this interval; or, the first binarization order corresponding to each network layer can be incremented from 1 according to the increasing order of the overall importance information. That is, according to the increasing order of the overall importance information, the first binarization order of the first network layer is 1, the first binarization order of the second layer is 2, and so on.

[0064] One way to determine the most important information is:

[0065] The target model is invoked to process the pre-prepared calibration data. During the processing, multiple network layers of the target model are connected in sequence. Each network layer calculates its input based on its own initial weight data and outputs the calculation result as the activation value of the network layer to the next network layer. The activation value output by the last network layer is used as the output probability distribution of the aforementioned target model.

[0066] By obtaining the activation values ​​of each network layer during the calibration data processing of the target model, the first importance information of the initial weight data contained in each network layer can be calculated based on these activation values. For specific calculation methods, please refer to relevant technologies, which will not be elaborated here.

[0067] Calibration data refers to the data used as input to the target model for processing to determine the first importance information. If the target model is a model for processing images, the calibration data can be any pre-selected image. If the target model is a model for processing text, the calibration data can be pre-edited text.

[0068] This embodiment does not limit the specific type of the first importance information. Any parameter in related technologies that can characterize the sensitivity of the corresponding initial weight data to the output probability distribution of the target model can be used as the first importance information.

[0069] As some examples, first importance information may include the covariance of the log-likelihood gradient, which is the gradient of the logarithm of the output probability distribution relative to the initial weight data. In this example, first importance information may also be referred to as Fisher information.

[0070] The value I(θ) in the i-th row and k-th column of the Fisher information matrix of a target model. ik It can be represented by the following formula (1).

[0071]

[0072] In the above formula, E θ This indicates the calculation of covariance, where P(x; θ) represents the output probability distribution calculated for the input calibration data x, log represents the logarithm to the base 10, and θ represents the output probability distribution. i and θ k These represent the initial weight data numbered i and k in the target model, respectively.

[0073] For any initial weight data θ i Let k equal i in formula (1), and obtain I(θ). ii This can be used as the primary information regarding the importance of the initial weight data.

[0074] S103, the corresponding initial weight data is binarized according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, so as to process the inference task based on the first quantized data.

[0075] In step S103, the initial weight data corresponding to each network layer can be binarized sequentially, and the binarization of the next network layer can be performed after the binarization of one network layer is completed. This can reduce the computing resources required for binarization.

[0076] The beneficial effects of this embodiment are as follows:

[0077] Based on the sensitivity of different initial weight data to the output probability distribution of the target model, a corresponding first binarization order is configured for different initial weight data. During binarization, initial weight data that is less sensitive to the output probability distribution can be processed into first quantized data with lower precision, thereby reducing the storage space occupied by this quantized data. On the other hand, initial weight data that is more sensitive to the output probability distribution can be processed into first quantized data with higher precision, avoiding a significant drop in the accuracy of the model processing results due to the low precision of this quantized data. Thus, this scheme can maintain the accuracy of the processing results obtained based on quantization to a certain extent while performing binarization processing on the initial weight data.

[0078] See Figure 2 For each set of initial weight data, the binarization process based on the first binarization order can be performed as follows:

[0079] S201, determine the average value of a set of initial weight data as the quantization mean.

[0080] S202, determine the binarization matrix and quantization scaling factor based on the initial weight data and quantization mean.

[0081] S203, if the number of binarized matrices obtained is less than the first binarization order, determine the new quantization mean, binarization matrix and quantization scaling factor based on the deviation of the estimated weight data from a set of initial weight data, until the number of binarized matrices corresponding to the first binarization order is obtained.

[0082] The estimated weights are determined based on the obtained quantization mean, binarization matrix, and quantization scaling factor.

[0083] The first quantization data includes at least one quantization mean, at least one quantization scaling factor, and at least one binarization matrix.

[0084] Press Figure 2 When the method shown performs binarization on a set of initial weight data, the quantization mean, scaling factor, and binarization matrix obtained in the first iteration can be denoted as first-order quantization data, the quantization mean, scaling factor, and binarization matrix obtained in the second iteration can be denoted as second-order quantization data, and so on, the quantization mean, scaling factor, and binarization matrix obtained in the Nth iteration can be denoted as N-order quantization data. The first quantization data corresponding to a set of initial weight data includes the corresponding first-order quantization data to N-order quantization data, where N is equal to the first binarization order corresponding to the set of initial weight data.

[0085] In step S201, the quantization mean can be determined on a per-weight matrix basis. If the initial weight data set corresponding to the network layer contains only the weight data of one weight matrix, the quantization mean corresponding to that weight matrix can be determined. If the initial weight data set corresponding to the network layer contains the weight data of multiple weight matrices, the quantization mean corresponding to each weight matrix can be determined, thus obtaining multiple quantization means.

[0086] A weight matrix corresponds to a quantized mean. The quantized mean of a weight matrix can be a column vector with the same number of rows as the weight matrix. The value of each row in the column vector is equal to the average of all weight data in the corresponding row of the weight matrix. For example, if the weight matrix is ​​a 512-row, 512-column matrix, the corresponding quantized mean can be a column vector with 512 rows. The value of the first row of the quantized mean is equal to the average of the 512 weight data in the first row of the weight matrix.

[0087] In S202, the binarization matrix and quantization scaling factor that minimize the weight data error can be determined based on the initial weight data and the quantization mean. The weight data error can be determined by the difference between the m-th estimated weight data and the initial weight data, or by the difference between the activation value calculated based on the m-th estimated weight data and the activation value calculated based on the initial weight data. Here, m represents the m-th determination of the binarization matrix and quantization scaling factor.

[0088] The initial set of weight data currently undergoing binarization can be represented by one or more weight matrices W, and the estimated weight data W for the m-th iteration... m 'It can be determined based on the quantization mean, binarization matrix and quantization scaling factor according to the following formula (2). m is greater than or equal to 1.

[0089]

[0090] Among them, a i B i and u i Let represent the quantization scaling factor determined in the i-th iteration, the binarization matrix determined in the i-th iteration, and the quantization mean determined in the i-th iteration, respectively.

[0091] When determining the binarization matrix and quantization scaling factor for the m-th time, the previously determined quantization scaling factor, binarization matrix, and quantization mean (when m is greater than 1), the quantization mean determined for the m-th time, and a pre-set initial value can be used. m and B m Substitute into formula (2), and then iteratively adjust a with the goal of minimizing the weighted data error. m and B m Finally, the m-th determination of the binarization matrix and quantization scaling factor is obtained.

[0092] A weight matrix corresponds to a quantization scaling factor. The quantization scaling factor for a weight matrix can be a column vector with the same number of rows as the weight matrix, and each row of the column vector contains a numerical value. For example, if the weight matrix is ​​a 512-row, 512-column matrix, the corresponding quantization scaling factor can be a column vector with 512 rows.

[0093] The number of rows and columns in the binarized matrix is ​​the same as that in the weight matrix W. Each value in the binarized matrix has only two possible values; for example, each value in the binarized matrix is ​​either 1 or -1. Furthermore, the values ​​in the binarized matrix correspond one-to-one with the weight data in the weight matrix W. Because each value has only two possible values, each value in the binarized matrix occupies only 1 bit in memory. A 512-row, 512-column binarized matrix occupies only 512*512 bits of storage space.

[0094] When calculating formula (2), B can be... i Each value in each row is multiplied by a. i Take the corresponding row's value, and then add u to the multiple products of each row. i The corresponding row value is used to obtain the corresponding calculation result.

[0095] In step S203, if the number of binarized matrices obtained is less than the first binarization order corresponding to the initial weight data of the current binarization process, then the quantization scaling factor, binarization matrix and quantization mean can be determined in the (m+1)th step.

[0096] In the (m+1)th iteration, the quantization scaling factor, binarization matrix, and quantization mean determined in the previous m iterations can be calculated using formula (2) to obtain the estimated weight data W for the mth iteration. m Then subtract the estimated weight data W_m from the initial weight data W_m. m The obtained matrix is ​​used as the difference weight data ΔW for the (m+1)th iteration. m+1 ΔW m+1 W and W have the same number of rows and columns, and the data they contain correspond one-to-one;

[0097] Then, the difference data ΔW from the (m+1)th iteration can be weighted according to the method for calculating the quantized mean in S201. m+1 The calculation is performed to obtain the quantization mean determined in the (m+1)th iteration. Then, following the method of S202, with the goal of minimizing the error of the weighted data, the binarization matrix and quantization scaling factor in the (m+1)th iteration are determined through iterative adjustment.

[0098] After obtaining the (m+1)th binarization matrix, quantization scaling factor, and quantization mean, if the number of binarization matrices obtained is still less than the corresponding first binarization order, the above process is repeated. If the number of binarization matrices obtained is equal to the corresponding first binarization order, the binarization process of the current set of initial weight data ends. The 1 to Nth order quantization data obtained at this point constitute the first quantization data corresponding to this set of initial weight data.

[0099] In some embodiments, the first quantization data may also exclude the quantization mean and only include the binarization matrix and quantization scaling factor. In this case, it can be... Figure 2 The operation related to determining the quantized mean in the corresponding method has been deleted; everything else remains unchanged.

[0100] When the first set of quantized data does not contain a quantized mean, the quantization scaling factor that needs to be determined each time can include column quantization scaling factor and row quantization scaling factor, the former and the latter... Figure 2 The meaning of the quantization scaling factor in the above context is the same; the latter can be a row vector, and the row quantization scaling factor determined for the i-th time can be denoted as b. i The number of columns is equal to the number of columns in the weight matrix W. Correspondingly, the estimated weight data for the m-th time can be represented by the following formula (3).

[0101]

[0102] When calculating formula (3), B can be...i Each value in the equation is multiplied by a. i The value in the corresponding row, multiplied by b. i The corresponding column values ​​are used to obtain the corresponding calculation results.

[0103] In some optional embodiments, when determining the binarization matrix and quantization scaling factor for the m-th time, an alternating refinement strategy can be used to alternately adjust the binarization matrix, quantization scaling factor, and quantization mean. The alternating refinement strategy refers to adjusting the value of one of the binarization matrix, quantization scaling factor, and quantization mean to minimize the weighted data error, while keeping the other two terms unchanged during the adjustment. This process is then repeated for the remaining two terms sequentially, and so on until the cycle stops. The stopping condition can be that the number of cycles reaches a set upper limit, or other conditions, which are not limited.

[0104] As an example, when determining the binarization matrix and quantization scaling factor for the m-th time, we can first keep the quantization mean obtained by calculating the average value and the quantization scaling factor set to the initial value unchanged, and adjust the binarization matrix B. m This minimizes the weighted data error; then, keeping the current quantization mean and binarization matrix unchanged, the quantization scaling factor 'a' is adjusted. m This minimizes the weighted data error; while keeping the current quantization scaling factor and binarization matrix unchanged, the quantization mean u is adjusted. m This minimizes the error in the weighted data; then, the cycle continues, adjusting the binarization matrix, quantization scaling factor, and quantization mean sequentially until the cycle stops, ultimately obtaining the determined binarization matrix, quantization scaling factor, and quantization mean for the m-th iteration.

[0105] In the above embodiments, the precision of the quantization scaling factor and the quantization mean can be the same as the precision of the initial weight data, or it can be lower than the precision of the initial weight data. For example, if the initial weight data is represented by a 32-bit floating-point number, the quantization scaling factor and the quantization mean can also be represented by a 32-bit floating-point number, or they can be represented by a lower-precision 8-bit integer (int).

[0106] After the initial weight data corresponding to each network layer in the first dataset is binarized in the manner described above, this first quantized data can be stored in the memory (e.g., hard disk) of the target device, enabling the target device to process inference tasks using this first quantized data. The target device refers to an electronic device that needs to use the target model to process inference tasks.

[0107] The target device's memory can store the first quantized data of each target network layer of the target model, and, if the target model has non-target network layers, can also store the initial weight data of the non-target network layers. The first quantized data of the target network layer is obtained by processing a set of initial weight data corresponding to the target network layer according to the aforementioned embodiment. The target network layer refers to the network layer that needs to be binarized, and the non-target network layer refers to the network that does not need to be binarized.

[0108] After obtaining the inference task to be processed, the processor of the target device loads each network layer from the memory in sequence. When loading a non-target network layer, the processor can load the initial weight data of the non-target network layer in the memory, use the initial weight data to calculate the input of the non-target network layer, and obtain the output of the non-target network layer. When loading the target network layer, the processor can calculate the estimated weight data of the target network layer based on the first quantization data of the target network layer in the memory according to formula (2), load the estimated weight data of the target network layer, calculate the input of the target network layer based on the estimated weight data of the target network layer, and obtain the output of the target network layer. This process is repeated until the output probability distribution of the target model is finally obtained, and the processing result is determined based on the output probability distribution.

[0109] As an example, see Figure 3 Assume the target model comprises four network layers, designated as network layer 1 to network layer 4. Network layers 1 and 4 are non-target network layers, while network layers 2 and 3 are target network layers. The target device's memory stores: a set of initial weight data corresponding to network layer 1, denoted as initial weight 1; a set of initial weight data corresponding to network layer 4, denoted as initial weight 4; the first quantized data corresponding to network layer 2, denoted as quantization weight 2; and the first quantized data corresponding to network layer 3, denoted as quantization weight 3.

[0110] The quantization weight 2 is obtained by binarizing a set of initial weight data corresponding to network layer 2 according to the aforementioned embodiment. The binarization order of the set of initial weight data corresponding to network layer 2 is 2. Therefore, the quantization data 2 includes 1st-order quantization data and 2nd-order quantization data determined sequentially. The 1st-order quantization data includes a1, B1 and u1, and the 2nd-order quantization data includes a2, B2 and u2.

[0111] The quantization weight 3 is obtained by binarizing a set of initial weight data corresponding to network layer 3 according to the aforementioned embodiment. The binarization order of the set of initial weight data corresponding to network layer 3 is 3. Therefore, the quantization data 3 includes 1st-order quantization data, 2nd-order quantization data, and 3rd-order quantization data determined sequentially. The composition of each order of quantization data is described above.

[0112] After obtaining the input data corresponding to the inference task (such as a piece of text data entered by a user), the target device first loads network layer 1 to process the task, reads the initial weight 1 from the memory, calculates the input data based on the initial weight 1, and obtains the output of network layer 1.

[0113] The target device loads network layer 2 to process the task, reads quantization weight 2 from the memory, calculates the corresponding estimated weight data according to the method of formula (2), and records it as estimated weight 2 (W2), that is, W2=a1*B1+u1+a2*B2+u2. The output of network layer 1 is calculated using estimated weight 2 to obtain the output of network layer 2.

[0114] The target device loads network layer 3 to process the task, reads quantization weight 3 from memory, calculates the corresponding estimated weight data using quantization weight 3, and denoted as estimated weight 3 (W3), i.e., W3=a1*B1+u1+a2*B2+u2+a3*B3+u3. The output of network layer 2 is calculated using estimated weight 3 to obtain the output of network layer 3.

[0115] The target device first loads network layer 4 to process the task, reads the initial weights 4 from the memory, calculates the output of network layer 3 based on the initial weights 4, obtains the output of network layer 4, which is the output probability distribution of the target model, and finally determines the processing result based on the output probability distribution.

[0116] As can be seen from the above storage method and the process of processing inference tasks, on the one hand, the target device can process the corresponding inference tasks based on the first quantized data corresponding to each network layer, so as to meet the usage needs of relevant users.

[0117] On the other hand, the first quantized data, which is obtained by binarizing a set of initial weight data and including quantization scaling coefficients, binarization matrix, and quantization mean, occupies less storage space than the corresponding set of initial weight data. Therefore, the method of this embodiment can reduce the storage resources consumed by deploying the target model on any device, so that the target model can be deployed on various electronic devices, especially on terminal devices with limited storage resources (such as laptops, home PCs, smartphones, etc.).

[0118] Based on the previous example, each initial weight data is a 32-bit floating-point number, so each initial weight data occupies 32 bits of storage space. A weight matrix consisting of a set of initial weight data with 512 rows and 512 columns (512*512) will occupy at least 512*512*32 bits of storage space.

[0119] Assuming the first binarization order of this initial set of weight data is 1, then the first quantized data obtained after the aforementioned processing contains only: a 512-row quantized mean column vector, occupying 512*32 bits of storage space (the quantized mean is stored as a 32-bit floating-point number), a 512-row quantized scaling factor column vector, occupying 512*32 bits of storage space (the quantized scaling factor is stored as a 32-bit floating-point number), and a binarized matrix occupying 512*512 bits. That is, the storage space occupied by the first quantized data is 512*32+512*32+512*512 bits, which is obviously less than 512*512*32 bits.

[0120] In some optional embodiments, the initial weight data contained in the first data set in S101 may not be the original weight data contained in the target model, but rather the weight data obtained after processing the original weight data.

[0121] For example, in some embodiments, the target model may include multiple sub-models with different processing capabilities;

[0122] Obtain the first dataset for the target model, including:

[0123] Obtain the weight data for multiple sub-models;

[0124] The weight data of multiple sub-models are merged and processed to obtain a first data set. The weight data contained in the first data set reflects the common data processing capabilities of multiple sub-models.

[0125] In the first dataset, each set of initial weight data is obtained by fusing data from the corresponding network layers of multiple sub-models;

[0126] The deviation between the weight data of the sub-model and the weight data contained in the first dataset is used as the residual weight data of the sub-model. The residual weight data reflects the unique data processing capability of the corresponding sub-model.

[0127] As examples, the target model described above could be a Mixture of Experts (MFA) model. A MFA model consists of multiple independent sub-models, each acting as an expert in a specific domain. Upon receiving input data, one or more sub-models process the data independently. Finally, the outputs of these sub-models are fused by a specific fusion module within the target model to obtain the output probability distribution of the MFA model.

[0128] A hybrid expert model can have multiple sub-models with the same structure, meaning each sub-model contains network layers. The network layers in different sub-models correspond one-to-one, and the number and size of the weight matrices between corresponding network layers are also the same. Therefore, the weight data of different sub-models can be considered to correspond one-to-one. For example, the weight data in the first row and first column of the weight matrix of network layer 1 in sub-model a corresponds to the weight data in the first row and first column of the weight matrix of network layer 1 in sub-model b. The weight data values ​​of different sub-models are different, and the differences in processing power between sub-models are reflected through these different weight data values.

[0129] Based on the above correspondence between different sub-models, during the fusion process, multiple corresponding weight data from multiple sub-models can be merged into one, and the fusion result can be used as an initial weight data.

[0130] Combination Figure 4 Assuming the target model includes sub-models a, b, and c, and a weight data a1 for sub-model a, a weight data b1 for sub-model b, and a weight data c1 for sub-model c correspond to each other, then a1, b1, and c1 are merged to obtain an initial weight data θ1. Similarly, the weight data a2, b2, and c2 belonging to the three sub-models are merged to obtain an initial weight data θ2.

[0131] By fusing the weight data of the same network layer in each sub-model based on the above correspondence, a set of initial weight data corresponding to that network layer can be obtained.

[0132] The initial weight data in the first dataset obtained in the above manner can reflect the common data processing capabilities of multiple sub-models, or in other words, can characterize the commonalities of multiple sub-models when processing data.

[0133] Optionally, the above fusion process can be performed only on the target network layer that needs binarization, or it can be performed on each network layer.

[0134] Given the first dataset obtained in the above manner, for each sub-model, the residual weight data corresponding to that sub-model can be further obtained based on the first dataset and the sub-model's own weight data, so as to utilize the unique data processing capabilities of each sub-model.

[0135] The precision of the residual weight data can be the same as that of the weight data of the sub-model itself, for example, both can be represented by 32-bit floating-point numbers, or it can be lower than the precision of the weight data of the sub-model itself.

[0136] The residual matrix formed by the residual weight data of the sub-model has the same structure as the weight matrix formed by the weight data of the sub-model itself. The residual weight data of the sub-model corresponds one-to-one with the weight data of the sub-model itself. Each residual weight data is obtained by subtracting the corresponding initial weight data in the first data set from the corresponding weight data in the sub-model.

[0137] The residual weight data of the sub-model can also be divided by network layer. For example, the residual weight data obtained by subtracting the initial weight data of the corresponding network layer 1 in the first data set from the set of weight data of the sub-model corresponding to the weight layer 1 is the residual weight data of the corresponding network layer 1 in the sub-model.

[0138] Combination Figure 4 For example, for sub-model a, the weight data a1 is subtracted from the corresponding initial weight data θ1 in the first data set, and the difference is used as the residual weight data corresponding to a1. a2 is subtracted from the initial weight data θ2, and the difference is used as the residual weight data corresponding to a2, and so on.

[0139] By fusing the weight data of multiple sub-models into a first data set, and then binarizing the first data set according to the aforementioned method, it is beneficial to reduce the amount of data that needs to be processed when performing binarization according to the aforementioned embodiment, thereby improving the efficiency of binarization processing.

[0140] In some embodiments, after fusing the weight data of multiple sub-models according to the above method and obtaining the residual weight data of each sub-model, the residual weight data of each sub-model can also be binarized, that is, the following steps can be performed:

[0141] The residual weight data of each sub-model is binarized to obtain the second quantized data corresponding to each sub-model;

[0142] The methods for binarizing the residual weight data and the methods for binarizing the initial weight data are different.

[0143] In this embodiment, the method for binarizing the residual weight data can be any quantization compression method in the field of model quantization compression that is different from the processing method in the previous embodiment. This embodiment does not limit the binarization processing method for the residual weight data.

[0144] As an example, for each sub-model, the residual weight data of the sub-model can be binarized by network layer as follows to obtain the second quantized data corresponding to each network layer of the sub-model.

[0145] For the network layer currently undergoing binarization, determine the maximum value r among all the residual weight data corresponding to that network layer.max and minimum value r min The difference between the two is used as the residual scaling factor s for that network layer, i.e., s = r max -r min ;

[0146] For each residual weight data x in the network layer, the residual weight data can be divided by the residual scaling factor. If the obtained ratio x / s is greater than 0, the residual quantization data corresponding to the residual weight data is determined to be 1. If the obtained ratio x / s is less than or equal to 0, the residual quantization data corresponding to the residual weight data is determined to be -1. After processing all the residual weight data in the network layer in this way, the binarization process of the network layer is completed. The obtained residual scaling factor and all residual quantization data constitute the second quantization data corresponding to the network layer. Since each residual quantization data can only take the values ​​1 and -1, it can occupy only 1 bit when stored. Therefore, the above binarization process can be called a 1-bit compression method. The matrix composed of the residual quantization data corresponding to a network layer can be used as the residual sign matrix of the network layer.

[0147] In the second quantization data, the residual scaling factor can be consistent with the precision of the residual weight data, for example, represented by a 32-bit floating-point number.

[0148] After binarizing the residual weight data of each sub-model using the above method, the target device can process the obtained inference task based on the first quantized data and the second quantized data corresponding to each sub-model.

[0149] The target device's memory can store the following data:

[0150] If the target model contains non-target network layers, then store the weight data of the corresponding non-target network layers in each sub-model;

[0151] For the target network layer, store the first quantized data corresponding to the target network layer, and the second quantized data corresponding to the target network layer for each sub-model.

[0152] When handling inference tasks, if you need to use a non-target network layer of a sub-model to process data, you can directly read the corresponding weight data and perform calculations based on that weight data. For details, please refer to [link to relevant documentation]. Figure 3 The part of the output of network layer 1 obtained based on the initial weight 1 in the corresponding embodiment will not be described in detail.

[0153] When processing inference tasks, if a target network layer of a sub-model needs to be used to process data, the first quantized data corresponding to that target network layer is read from memory and processed according to... Figure 3In the corresponding embodiment, the method for calculating the estimated weight data calculates the first quantized data to obtain the estimated weight data corresponding to the target network layer.

[0154] Furthermore, the second quantized data corresponding to the target network layer and belonging to the sub-model is read, and the residual quantized data therein is multiplied by the residual scaling factor to obtain the estimated residual data corresponding to each residual quantized data.

[0155] Finally, the corresponding estimated weight data and estimated residual data are added together, and the result is used as the restored weight data of the target network layer of the sub-model. Based on the restored weight data, the input of the sub-model to the target network layer is calculated, and the output of the sub-model to the target network layer can be obtained.

[0156] In this embodiment, the weight data of multiple sub-models can be fused and processed using various methods. For example, the corresponding weight data in all sub-models can be summed directly, and the result can be used as the initial weight data in the first data set. Alternatively, the average value of the corresponding weight data in all sub-models can be calculated, and the result can be used as the initial weight data in the first data set.

[0157] In some embodiments, see Figure 5 Alternatively, the weight data of multiple sub-models can be merged and processed in the following way to obtain the first data set.

[0158] S501, determine the second importance information for each sub-model. The second importance information characterizes the sensitivity of the sub-model's weight data to the output probability distribution of the target model.

[0159] S502, the weight data of multiple sub-models are fused according to the second importance information to obtain the first data set.

[0160] In step S501, for each sub-model, the second importance information of each weight data corresponding to each target network layer in the sub-model can be calculated based on the calibration data. The second importance information can be a parameter of the same type as the first importance information. For example, the second importance information of the weight data in the sub-model can be the Fisher information of the weight data.

[0161] The method for obtaining the second importance information of the weight data in the sub-model can refer to the method for obtaining the first importance information in the foregoing embodiments, and will not be repeated here.

[0162] In S502, multiple weight data corresponding to each other in multiple sub-models can be weighted and summed according to their respective second importance information, and the result can be used as the corresponding initial weight data in the first data set.

[0163] by Figure 4For example, the corresponding weight data a1, b1 and c1 in the three sub-models can be merged into the initial weight data θ1 according to the following formula (4).

[0164]

[0165] Where Fa1 represents the second importance information corresponding to a1, Fb1 represents the second importance information corresponding to b1, and Fc1 represents the second importance information corresponding to c1. In some embodiments, Fa1, Fb1, and Fc1 in formula (4) can also be replaced by the ratio of Fa1 to S1, the ratio of Fb1 to S1, and the ratio of Fc1 to S1, respectively, where S1 is the sum of the three second importance information, i.e., S1 = Fa1 + Fb1 + Fc1.

[0166] The advantages of fusing the weight data of multiple sub-models using the above method are:

[0167] For each sub-model, the correlation between its own weight data and the data processing capabilities shared by multiple sub-models varies. By fusing the weight data of each sub-model based on the second importance information of each sub-model, the difference in the correlation between different sub-models can be taken into account during the fusion process. In this way, the initial weight data obtained can more accurately reflect the data processing capabilities shared by multiple sub-models.

[0168] according to Figure 5 When the first data set is obtained by fusing the methods, the first importance information corresponding to each initial weight data in the first data set can still be calculated based on the calibration data according to the method of the aforementioned embodiment. Alternatively, the second importance information of the weight data of each sub-model corresponding to this initial weight data can be added or averaged, and the result can be used as the first importance information of this initial weight data.

[0169] Based on the previous example, for the initial weight data θ1 obtained by fusing weight data a1, b1 and c1, its corresponding first importance information can be equal to (Fa1+Fb1+Fc1) / 3.

[0170] In some optional embodiments, the residual weight data corresponding to each sub-model can also be binarized in the same way as the initial weight data binarization in the aforementioned embodiments.

[0171] For the residual weight data corresponding to different network layers in the sub-model, the binarization order of the residual weight data of different network layers can be determined according to the second importance information of the weight data corresponding to that network layer. If the second importance information indicates that the weight data of a network layer of a sub-model is more sensitive to the output probability distribution of the target model, a higher binarization order can be assigned to the residual weight data of the corresponding network layer of this sub-model. If the second importance information indicates that the weight data of a network layer of a sub-model is less sensitive to the output probability distribution of the target model, a lower binarization order can be assigned to the residual weight data of the corresponding network layer of this sub-model.

[0172] Then, according to the assigned binarization order, and based on the method for binarizing the initial weight data in the aforementioned embodiment, the residual weight data corresponding to different network layers in each sub-model are binarized to obtain the second quantized data corresponding to each network layer of each sub-model.

[0173] In this case, the method for calculating the estimated residual data based on the second quantization data can be referred to the method for calculating the estimated weight data based on the first quantization data in the foregoing embodiments, and will not be repeated here.

[0174] In some alternative embodiments, the first dataset of the target model can be obtained in the following ways:

[0175] Identify at least one target network layer among multiple network layers of the target model;

[0176] The initial weight data of the corresponding target network layer is obtained as the first data set of the target model.

[0177] In other words, the binarization method described in the above embodiments can be applied only to the initial weight data of some network layers (i.e., the target network layer) in the target model, and the initial weight data of non-target network layers other than the target network layer can be exempted from binarization.

[0178] There are no restrictions on the strategy for selecting the target network layer. For example, the target network layer can be selected based on the number of corresponding initial weight data. That is, among multiple network layers, at least a preset number of network layers can be selected as the target network layer according to the number of corresponding initial weight data.

[0179] Alternatively, the target network layer can be selected based on its role in the target model. For example, the network layers of the target model can generally be divided into three categories: transformation layers, embedding layers, and output head layers. Each category can have one or more network layers. The embedding layer is used to transform the input data into a form suitable for processing by the transformation layer. The transformation layer is used to perform inference calculations on the input. The output head layer is used to encode the output of the transformation layer into the output probability distribution of the target model. In this embodiment, all transformation layers of the target model can be determined as target network layers, while the embedding layer and output head layer are designated as non-target network layers.

[0180] Optionally, the corresponding initial weight data is binarized according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, including:

[0181] The weighted smoothing coefficients are determined based on the calibration data used for binarization processing;

[0182] The initial weight data is scaled according to the weight smoothing coefficient to obtain scaled weight data;

[0183] The scaled weight data is binarized according to the first binarization order corresponding to the initial weight data to obtain the first quantized data corresponding to the initial weight data.

[0184] One method for scaling the initial weight data based on the weight smoothing coefficient is to process the initial weight data with the corresponding weight smoothing coefficient, and use the result as the scaled weight data.

[0185] Different initial weight data in the first dataset can correspond to the same or different weight smoothing coefficients. As examples, a single weight smoothing coefficient can be determined for the entire first dataset, and all initial weight data in the first dataset are scaled using the same weight smoothing coefficient; alternatively, a separate weight smoothing coefficient can be determined for the initial weight data corresponding to each network layer, with different weight smoothing coefficients corresponding to the initial weight data of different network layers in the first dataset; or, the weight smoothing coefficient can be determined by the rows of the weight matrix, meaning that initial weight data belonging to the same row of the same weight matrix correspond to the same weight smoothing coefficient, with each row of each weight matrix in each network layer corresponding to a weight smoothing coefficient, and each row of the weight matrix equivalent to a channel.

[0186] When determining a weight smoothing coefficient for the entire first dataset, the weight smoothing coefficient can be determined in the following ways:

[0187] Before inputting calibration data into the target model for processing, the calibration data needs to be converted into a vector or matrix that the network layer of the target model can process. Therefore, the average value or the sum of all values ​​contained in the vector or matrix after the calibration data is converted can be calculated, and the calculation result can be used as the weight smoothing coefficient.

[0188] When determining the weight smoothing coefficient for each network layer's initial weight data, the method for determining the weight smoothing coefficient can be as follows:

[0189] The target model is invoked to process the calibration data. As explained earlier, during the calibration data processing, each network layer receives the activation values ​​output by the previous network layer as input. These activation values ​​are in vector or matrix form. Therefore, for each network layer, the average or sum of all data contained in the activation values ​​input to that network layer can be calculated, and the result can be used as the weight smoothing coefficient corresponding to that network layer. The input to the first network layer of the target model is the vector or matrix obtained by transforming the calibration data; therefore, the weight smoothing coefficient corresponding to the first network layer can be determined using the same method as before.

[0190] When determining the weight smoothing coefficient based on the row units of the weight matrix, the weight smoothing coefficient can be determined in the following ways:

[0191] When the activation values ​​of the input network layer are vectors, the number of data points in the vector is generally the same as the number of rows in the weight matrix of the network layer. For example, if the weight matrix of the network layer is 512 rows and 512 columns, then the input activation values ​​are generally 512-dimensional vectors, and the 512 data points can correspond to the 512 rows of the weight matrix. In this case, the data point in the activation values ​​corresponding to a row of the weight matrix can be used as the weight smoothing coefficient corresponding to the initial weight data of that row. For example, the weight smoothing coefficient used for the initial weight data in the 32nd row of the weight matrix can be the 32nd data point in the activation values.

[0192] When the activation values ​​input to a network layer are a matrix, the number of columns in this matrix is ​​generally the same as the number of rows in the weight matrix of the network layer. For example, if the weight matrix of the network layer has 512 rows, then the input activation values ​​are a matrix with 512 columns. In this case, the average or sum of the data in each column of the activation values ​​can be calculated, and the result can be used as the weight smoothing coefficient for the initial weight data in the corresponding row of the weight matrix. For example, the weight smoothing coefficient used for the initial weight data in the 32nd row of the weight matrix can be the average of all the data in the 32nd column of the activation values.

[0193] The above methods for determining the weight smoothing coefficient are only some optional examples. In actual applications, the weight smoothing coefficient can be determined in different ways and at different levels, without limitation.

[0194] Optionally, if the scaling is performed first based on the weight smoothing coefficient before binarization, the first quantized data may also include the weight smoothing coefficient used. When the target device needs to calculate the estimated weight data to process the inference task, the estimated weight data calculated according to the aforementioned embodiment can be multiplied by the corresponding weight smoothing coefficient and then used to process the inference task.

[0195] The method for binarizing the scaling weight data is the same as the method for binarizing the initial weight data in the previous embodiments, and will not be described again.

[0196] The advantage of scaling the initial weight data according to the weight smoothing coefficient is that:

[0197] Scaling the initial weight data based on the weight smoothing coefficient can compress the range of weight data values ​​and reduce outliers with excessively large values ​​in the scaled weight data used for binarization. This helps to further reduce weight data errors during subsequent binarization. The smaller the weight data error, the more accurate the processing results obtained by the target device in the inference task based on the first quantized data.

[0198] Optionally, the corresponding initial weight data is binarized according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, including:

[0199] For each set of initial weight data, the weight data of different parts in the set of initial weight data are binarized according to the third binarization order and the first binarization order corresponding to the set of initial weight data to obtain the first quantized data corresponding to the set of initial weight data.

[0200] The first and third binarization orders are different, meaning that different parts of the initial weight data have different degrees of influence on the processing results of the target model.

[0201] When binarizing a set of initial weight data, the third binarization order used can be less than the first binarization order corresponding to the set of initial weight data determined based on the first importance information in the aforementioned embodiment. Accordingly, the influence of the weight data binarized according to the third binarization order is weaker, while the influence of the weight data binarized according to the first binarization order is stronger.

[0202] In this embodiment, a set of initial weight data corresponding to the same network layer can be divided into two parts according to the columns of the weight matrix. That is, at least one column of weight data whose influence on the processing result is greater than a certain threshold (indicating a strong influence) is identified and recorded as significant weight data. At least one column of weight data other than significant weight data is regarded as non-significant weight data with a weaker influence.

[0203] Then, based on the first binarization order corresponding to this set of initial weight data, the significant weight data can be binarized, and a third binarization order less than the first binarization order can be determined (for example, the first binarization order is reduced by 1 or 2). The non-significant weight data can then be binarized based on the third binarization order.

[0204] The methods for binarizing significant weight data based on the first binarization order and for binarizing non-significant weight data based on the third binarization order can be found in the aforementioned embodiment of binarizing initial weight data based on the first binarization order, and will not be repeated here.

[0205] One way to determine the degree of influence of initial weight data on the processing results is as follows:

[0206] For any network layer, the Hessian matrix of the initial weight data corresponding to that network is calculated based on the calibration data. The influence index of the weight data in different columns of the corresponding weight matrix is ​​then determined based on the Hessian matrix. The methods for calculating the Hessian matrix and determining the influence index based on the Hessian matrix can be found in relevant techniques and will not be elaborated here.

[0207] The advantage of performing binarization in the above manner is that:

[0208] It can further reduce the binarization order of the initial weight data, which has a weaker impact on the processing results, thereby reducing the amount of data in the first quantized data after binarization and saving the storage space occupied by storing the first quantized data.

[0209] This application also provides a data processing apparatus; please refer to [link to relevant documentation]. Figure 6 This is a schematic diagram of the structure of the device, which may include the following units.

[0210] The obtaining unit 601 is used to obtain a first data set of the target model. The first data set includes at least one set of initial weight data. Each set of initial weight data corresponds to a network layer of the target model. The target model is a large language model used to process inference tasks.

[0211] The determining unit 602 is used to determine the first binarization order corresponding to the initial weight data based on the first importance information of the initial weight data. The first importance information characterizes the sensitivity of the corresponding initial weight data to the output probability distribution of the target model. The output probability distribution is used to determine the processing result of the target model.

[0212] The processing unit 603 is used to perform binarization processing on the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, so as to process the inference task based on the first quantized data.

[0213] Optionally, the first importance information includes the covariance of the log-likelihood gradient, which is the gradient of the logarithm of the output probability distribution relative to the initial weight data.

[0214] Optionally, the processing unit 603 performs binarization processing on the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, including:

[0215] Determine the average value of a set of initial weighted data as the quantization mean;

[0216] Based on the initial weight data and the quantization mean, determine the binarization matrix and quantization scaling factor;

[0217] If the number of binarized matrices obtained is less than the first binarization order, a new quantization mean, binarization matrix, and quantization scaling factor are determined based on the deviation of the estimated weight data from a set of initial weight data, until the number of binarized matrices corresponding to the first binarization order is obtained; the estimated weight data is determined based on the obtained quantization mean, binarization matrix, and quantization scaling factor.

[0218] The first quantization data includes at least one quantization mean, at least one quantization scaling factor, and at least one binarization matrix.

[0219] Optionally, the target model may include multiple sub-models with different processing capabilities;

[0220] Unit 601 obtains the first data set of the target model, including:

[0221] Obtain the weight data for multiple sub-models;

[0222] The weight data of multiple sub-models are merged and processed to obtain a first data set. The weight data contained in the first data set reflects the common data processing capabilities of multiple sub-models.

[0223] In the first dataset, each set of initial weight data is obtained by fusing data from the corresponding network layers of multiple sub-models;

[0224] The deviation between the weight data of the sub-model and the weight data contained in the first dataset is used as the residual weight data of the sub-model. The residual weight data reflects the unique data processing capability of the corresponding sub-model.

[0225] Optionally, the weight data of multiple sub-models are fused and processed by unit 601 to obtain a first data set, including:

[0226] Determine the second importance information for each sub-model. The second importance information characterizes the sensitivity of the sub-model's weight data to the output probability distribution of the target model.

[0227] The weight data of multiple sub-models are fused based on the second importance information to obtain the first data set.

[0228] Optionally, the processing unit 603 can also be used for:

[0229] The residual weight data of each sub-model is binarized to obtain the second quantized data corresponding to each sub-model;

[0230] The methods for binarizing the residual weight data and the methods for binarizing the initial weight data are different.

[0231] Optionally, the obtaining unit 601 obtains a first data set of the target model, including:

[0232] Identify at least one target network layer among multiple network layers of the target model;

[0233] The initial weight data of the corresponding target network layer is obtained as the first data set of the target model.

[0234] Optionally, the processing unit 603 performs binarization processing on the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, including:

[0235] The weighted smoothing coefficients are determined based on the calibration data used for binarization processing;

[0236] The initial weight data is scaled according to the weight smoothing coefficient to obtain scaled weight data;

[0237] The scaled weight data is binarized according to the first binarization order corresponding to the initial weight data to obtain the first quantized data corresponding to the initial weight data.

[0238] Optionally, the processing unit 603 performs binarization processing on the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, including:

[0239] For each set of initial weight data, the weight data of different parts in the set of initial weight data are binarized according to the third binarization order and the first binarization order corresponding to the set of initial weight data to obtain the first quantized data corresponding to the set of initial weight data.

[0240] The first and third binarization orders are different, meaning that different parts of the initial weight data have different degrees of influence on the processing results of the target model.

[0241] The working principle of the data processing device in this embodiment can be found in the relevant steps of the data processing method in the foregoing embodiment, and will not be repeated here.

[0242] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0243] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0244] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0245] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0246] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, comprising: A first dataset of the target model is obtained. The first dataset includes at least one set of initial weight data. Each set of initial weight data corresponds to a network layer of the target model. The target model is a large language model for processing inference tasks. The first binarization order corresponding to the initial weight data is determined based on the first importance information of the initial weight data. The first importance information characterizes the sensitivity of the corresponding initial weight data to the output probability distribution of the target model. The output probability distribution is used to determine the processing result of the target model. The initial weight data is binarized according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, so as to process the inference task based on the first quantized data.

2. The method according to claim 1, wherein the first importance information includes the covariance of the log-likelihood gradient, the log-likelihood gradient being the gradient of the logarithm of the output probability distribution relative to the initial weight data.

3. The method according to claim 1, wherein binarizing the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data includes: The average value of the initial weight data set is determined as the quantization mean; Based on the initial weight data and the quantization mean, determine the binarization matrix and the quantization scaling factor; If the number of binarized matrices obtained is less than the first binarization order, a new quantization mean, binarization matrix, and quantization scaling factor are determined based on the deviation of the estimated weight data relative to a set of initial weight data, until a number of binarized matrices corresponding to the first binarization order are obtained. The estimated weight data is determined based on the obtained quantization mean, binarization matrix, and quantization scaling factor; The first quantization data includes at least one quantization mean, at least one quantization scaling factor, and at least one binarization matrix.

4. The method according to claim 1, wherein the target model comprises multiple sub-models with different processing capabilities; The first data set for obtaining the target model includes: Obtain the weight data of multiple sub-models; The weight data of the multiple sub-models are fused and processed to obtain a first data set, wherein the weight data contained in the first data set reflects the common data processing capabilities of the multiple sub-models; Wherein, each set of initial weight data in the first data set is obtained by fusing data from the corresponding network layers contained in the multiple sub-models; The deviation between the weight data of the sub-model and the weight data contained in the first dataset is used as the residual weight data of the sub-model, and the residual weight data reflects the unique data processing capability of the corresponding sub-model.

5. The method according to claim 4, wherein fusing the weight data of the plurality of sub-models to obtain a first data set includes: Determine a second importance information for each of the sub-models, the second importance information characterizing the sensitivity of the weight data of the sub-model to the output probability distribution of the target model; The weight data of multiple sub-models are fused based on the second importance information to obtain a first data set.

6. The method according to claim 4, further comprising: The residual weight data of each sub-model is binarized to obtain the second quantized data corresponding to each sub-model; The method for binarizing the residual weight data is different from the method for binarizing the initial weight data.

7. The method according to claim 1, wherein obtaining the first data set of the target model comprises: Identify at least one target network layer among multiple network layers of the target model; The initial weight data corresponding to the target network layer is obtained as the first data set of the target model.

8. The method according to claim 1, wherein binarizing the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data comprises: The weighted smoothing coefficients are determined based on the calibration data used for binarization processing; The initial weight data is scaled according to the weight smoothing coefficient to obtain scaled weight data; The scaling weight data is binarized according to the first binarization order corresponding to the initial weight data to obtain the first quantized data corresponding to the initial weight data.

9. The method according to claim 1, wherein binarizing the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data comprises: For each set of initial weight data, the weight data of different parts of the set of initial weight data are binarized according to the third binarization order and the first binarization order corresponding to the set of initial weight data to obtain the first quantized data corresponding to the set of initial weight data. The first binarization order and the third binarization order are different, and the influence of different parts of the weight data in the initial set of weight data on the processing result of the target model is different.

10. A data processing apparatus, comprising: The obtaining unit is used to obtain a first data set of the target model. The first data set includes at least one set of initial weight data. Each set of initial weight data corresponds to a network layer of the target model. The target model is a large language model for processing inference tasks. The determining unit is configured to determine the first binarization order corresponding to the initial weight data based on the first importance information of the initial weight data, wherein the first importance information characterizes the sensitivity of the corresponding initial weight data to the output probability distribution of the target model, and the output probability distribution is used to determine the processing result of the target model; The processing unit is configured to perform binarization processing on the corresponding initial weight data according to the first binarization order to obtain the first quantized data corresponding to the initial weight data, so as to process the inference task based on the first quantized data.

Images