Large model quantization method, large model quantization device and electronic equipment

Abstract

The embodiment of the invention provides a large model quantification method, a large model quantification device and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: respectively carrying out scaling processing on a first tensor and a second tensor in an attention mechanism module in a large model to obtain a scaled first tensor and a scaled second tensor; performing quantization processing on the scaled first tensor and the scaled second tensor respectively to obtain a quantized first tensor and a quantized second tensor; wherein the matrix product of the quantized first tensor and the quantized second tensor is used for determining the attention score tensor of the attention mechanism module. According to the embodiment of the invention, the reasoning speed can be improved while the model reasoning precision is guaranteed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a large model quantization method, a large model quantization device, and an electronic device. Background Technology

[0002] Large models refer to machine learning models with a large number of parameters and complex computational structures. Attention mechanisms are a technique in deep learning that allows models to focus on specific parts of an input sequence when processing sequential data. In some large models, such as those based on the Transformer architecture, the attention mechanism module is an important component of the model structure.

[0003] To improve the inference speed of such large models, the attention mechanism model can be quantized. Summary of the Invention

[0004] This application provides a large model quantization method, a large model quantization device, and an electronic device. By quantizing at least the attention mechanism module in the model structure, the inference speed is improved while ensuring the model inference accuracy.

[0005] In a first aspect, embodiments of this application provide a large model quantization method, wherein the large model includes an attention mechanism module. The method includes: scaling a first tensor and a second tensor in the attention mechanism module to obtain scaled first tensor and scaled second tensor; and quantizing the scaled first tensor and the scaled second tensor to obtain quantized first tensor and quantized second tensor; wherein the matrix product of the quantized first tensor and the quantized second tensor is used to determine the attention score tensor of the attention mechanism module.

[0006] In conjunction with the first aspect, in the first implementation of the first aspect, the aforementioned large model quantization method further includes: when outliers exist in the channel dimensions of at least one of the first tensor and the second tensor, determining the i-th first target value based on the value of the element in the i-th channel of the first tensor, where i is a positive integer not greater than the total number of channels of the first tensor; determining the i-th second target value based on the value of the element in the i-th channel of the second tensor; and determining the i-th scaling scale based on the i-th first target value, the i-th second target value, and the i-th scaling factor corresponding to the i-th channel; wherein the i-th scaling scale is used to scale the i-th channel in the first tensor and the i-th channel in the second tensor respectively.

[0007] The above implementation provides a method for determining the i-th scaling scale. The scaling scale determined based on the first and second target values ​​reflects the data characteristics of the two tensors in the current channel (the i-th channel). Scaling the i-th channel using this scaling scale helps to make the numerical distribution of the two tensors in the current channel more uniform, thereby improving the quantization accuracy of the tensors. Furthermore, the i-th scaling scale is used for scaling the i-th channel, meaning that different channels have different scaling scales. Therefore, even if outliers in the first and second tensors are not in the same channel, outliers appearing in different channels can be processed separately. It is evident that the solution provided in this application, through refined scaling, facilitates targeted scaling of each channel, resulting in a more uniform numerical distribution in each channel after scaling.

[0008] In conjunction with the first implementation of the first aspect, in the second implementation, determining the i-th first target value based on the values ​​of the elements in the i-th channel of the first tensor includes: determining the maximum value among the absolute values ​​of all elements in the i-th channel of the first tensor as the i-th first target value; or, determining the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​among all elements in the i-th channel of the first tensor as the i-th first target value.

[0009] The above implementation provides a method for determining the first target value. The first target value and the second target value will be used to determine the scaling scale corresponding to the current channel. Specifically, the target value is the maximum absolute value or the statistical value of the N largest absolute values ​​in the current channel. It can be seen that if there are outliers in the current channel, determining the scaling scale through the target value can identify the characteristics of the outliers in the channel. Therefore, the target value helps to improve the accuracy of the determined scaling scale, thereby facilitating more precise scaling processing.

[0010] In conjunction with the first implementation of the first aspect, in the third implementation, determining the i-th scaling scale based on the i-th first target value, the i-th second target value, and the i-th scaling factor corresponding to the i-th channel includes: determining the i-th first value based on the i-th first target value and the i-th scaling factor; determining the i-th second value based on the i-th second target value and the i-th scaling factor; and determining the i-th scaling scale based on the i-th first value and the i-th second value; wherein the i-th scaling scale is used to scale the i-th channel of the first tensor and the i-th channel of the second tensor.

[0011] The above implementation provides a method for determining the i-th scaling factor. The first and second target values ​​are jointly constructed using the i-th scaling factor, respectively, to obtain the corresponding first and second values. By applying the i-th scaling factor to the target values ​​from the i-th channels of the first and second tensors, the target values ​​from both tensors are processed synchronously. The scaling factor determined by the first and second values ​​helps to promote a more uniform numerical distribution in the current channels of the two tensors, thereby improving the quantization accuracy of the tensors.

[0012] In conjunction with the third implementation of the first aspect, in the fourth implementation, the above-mentioned determination of the i-th scaling scale based on the i-th first value and the i-th second value includes: determining the ratio of the i-th first value and the i-th second value as the i-th scaling scale;

[0013] The scaling process performed on the first and second tensors in the attention mechanism module to obtain the scaled first and second tensors includes: comparing the element in the i-th channel of the first tensor with the i-th scaling scale to obtain the i-th channel in the scaled first tensor; and multiplying the element in the i-th channel of the second tensor with the i-th scaling scale to obtain the i-th channel in the scaled second tensor.

[0014] The above implementation provides an example of scaling the i-th channel of two tensors by the i-th scaling scale, where the i-th scaling scale is determined by the ratio of the first value and the second value.

[0015] In conjunction with the first implementation of the first aspect, in the fifth implementation, the method further includes: determining M candidate values ​​for the i-th scaling factor, where M is an integer greater than 1; determining a target candidate value for the i-th scaling scale based on the target candidate value of the i-th scaling factor, the i-th first target value, and the i-th second target value; obtaining the predicted value of the i-th channel in the first tensor and the predicted value of the i-th channel in the second tensor under the target candidate value of the i-th scaling scale; determining target prediction information based on the predicted values ​​of the i-th channels in the first and second tensors obtained under the target candidate value of the i-th scaling scale; determining actual information based on the i-th channel in the first and second tensors; determining loss information based on the target prediction information and the actual information; and determining the candidate value corresponding to the minimum value among the M loss information as the i-th scaling factor.

[0016] The above implementation provides a method for determining the i-th scaling factor. The i-th scaling factor is used to determine the scaling scale corresponding to the current channel. That is, different channels use different scaling factors. By finely determining the scaling factor for each channel, it is beneficial to the uniformity of the distribution of each channel after scaling. Specifically, on the one hand, the actual information is determined based on the original values ​​of the two tensors in the i-th channel before scaling. On the other hand, the predicted information is determined based on the predicted values ​​of the two tensors in the i-th channel after scaling and quantization. Then, the optimal scaling factor is determined based on the loss between the predicted information and the actual information. Since this optimal scaling factor is determined based on the values ​​of the two tensors in the current channel, it can reflect the numerical characteristics of the current channel. Therefore, scaling the i-th channel using the i-th scaling scale determined by this optimal scaling factor is beneficial to the uniformity of the numerical distribution of the current channel.

[0017] In conjunction with the first aspect, in the sixth implementation of the first aspect, when outliers exist in the channel dimensions of at least one of the first tensor and the second tensor, the method further includes: determining an i-th first target value based on the value of the element in the i-th channel of the first tensor, where i is a positive integer not greater than the total number of channels of the first tensor; determining an i-th second target value based on the value of the element in the i-th channel of the second tensor; and determining an i-th scaling scale based on the i-th first target value, the i-th second target value, and a scaling factor; wherein the i-th scaling scale is used to scale the i-th channel of the first tensor and the second tensor respectively.

[0018] The above implementation provides a method for determining the i-th scaling scale. The scaling scale determined based on the first and second target values ​​reflects the data characteristics of the two tensors in the current channel (the i-th channel). Scaling the i-th channel using this scaling scale helps to make the numerical distribution of the two tensors in the current channel more uniform, thereby improving the quantization accuracy of the tensors. Furthermore, the i-th scaling scale is used for scaling the i-th channel, meaning that different channels have different scaling scales. Therefore, even if outliers in the first and second tensors are not in the same channel, outliers appearing in different channels can be processed separately. It is evident that the solution provided in this application, through refined scaling, facilitates targeted scaling of each channel, resulting in a more uniform numerical distribution in each channel after scaling.

[0019] In conjunction with the sixth implementation, the seventh implementation of the first aspect further includes: determining M candidate values ​​for the scaling factor, where M is an integer greater than 1; determining a target candidate value for the i-th scaling scale based on the target candidate value of the scaling factor, the first target value corresponding to the first tensor, and the second target value corresponding to the second tensor; obtaining a predicted first tensor corresponding to the first tensor and a predicted second tensor corresponding to the second tensor under the target candidate value of the i-th scaling scale; determining target prediction information based on the predicted first tensor and the predicted second tensor obtained under the target candidate value of the i-th scaling scale; determining actual information based on the first tensor and the second tensor; determining loss information based on the target prediction information and the actual information; and determining the candidate value corresponding to the minimum value among the M loss information as the scaling factor.

[0020] The above implementation provides a method for determining the scaling factor. This scaling factor is applied to all channels of the first and second tensors, meaning that the same scaling factor is used for different channels. Determining the scaling factor for multiple channels in the tensor using the same scaling factor helps to improve scaling efficiency while ensuring a relatively uniform distribution after scaling. Specifically, on the one hand, the actual information is determined based on the original values ​​of the two tensors before scaling; on the other hand, the predicted information is determined based on the predicted values ​​of the two tensors after scaling and quantization. Then, the optimal scaling factor is determined based on the loss between the predicted and actual information. Since this optimal scaling factor is determined based on the values ​​of the two tensors, it reflects the overall characteristics of the tensor's channel values. Therefore, scaling the i-th channel using the i-th scaling scale determined by this optimal scaling factor ensures a relatively uniform distribution after scaling while also improving scaling efficiency.

[0021] In combination with the first aspect and any of the first to seventh implementations mentioned above, in the eighth implementation of the first aspect, the first tensor mentioned above is a query tensor, and the second tensor mentioned above is a key tensor;

[0022] The above-mentioned quantization processing of the scaled first tensor and the scaled second tensor to obtain the quantized first tensor and the quantized second tensor includes: quantizing the scaled first tensor and the scaled second tensor according to the token per_token to obtain the quantized first tensor and the quantized second tensor; or quantizing the scaled first tensor and the scaled second tensor according to the tensor per_tensor to obtain the quantized first tensor and the quantized second tensor.

[0023] The above implementation provides a quantization method for query tensors and key tensors in the attention mechanism. Since the tensors have undergone the above scaling process, the numerical distribution of query tensors and key tensors is more uniform, which can better utilize the limited quantization levels in the above quantization method and improve the accuracy of the quantized model.

[0024] In conjunction with the first aspect and any of the first to seventh implementations mentioned above, in the ninth implementation of the first aspect, the first tensor is an attention score distribution tensor, and the second tensor is a value tensor, wherein the attention score distribution tensor is determined based on the attention score tensor; the quantization processing of the scaled first tensor and the scaled second tensor to obtain the quantized first tensor and the quantized second tensor respectively includes: performing per_tensor quantization processing on the scaled first tensor to obtain the quantized first tensor; and performing per_token quantization processing on the scaled second tensor to obtain the quantized second tensor.

[0025] In conjunction with the first aspect and any of the first to seventh implementations described above, in the ninth implementation of the first aspect, the method further includes: determining the smoothing scale corresponding to the linear layer in the attention mechanism module; smoothing the input and weights of the linear layer in the attention mechanism module according to the smoothing scale; and quantizing the smoothed input and weights.

[0026] The above implementation provides an example of smooth quantization processing for the linear transformation layer in the attention mechanism, which is beneficial to further improve the model's inference speed and accuracy.

[0027] In conjunction with the ninth implementation of the first aspect, the tenth implementation of the first aspect further includes a linear layer in the large model, and the method further includes: determining the smoothing scale corresponding to the j-th linear layer, where j is a positive integer; smoothing the input and weights of the j-th linear layer according to the smoothing scale; and quantizing the smoothed input and weights of the j-th linear layer; wherein the smoothing scales of the input projection and the gated projection in the attention mechanism module are equal.

[0028] The above implementation provides an example of smooth quantization processing for other linear layers in a large model, in addition to the quantization processing of the attention mechanism module in the large model, which is beneficial to further improve the inference speed and accuracy of the model.

[0029] In conjunction with the tenth implementation of the first aspect, in the eleventh implementation of the first aspect, if the model structure before the above linear layer is a normalized layer, the above method further includes: smoothing the weights of the above normalized layer according to the smoothing scale of the current linear layer.

[0030] Secondly, embodiments of this application provide a large model quantization device, wherein the large model includes an attention mechanism module, and the device includes: a scaling processing module and a quantization processing module;

[0031] The scaling processing module is used to scale the first tensor and the second tensor in the attention mechanism module to obtain scaled first tensor and scaled second tensor respectively; the quantization processing module is used to quantize the scaled first tensor and the scaled second tensor to obtain quantized first tensor and quantized second tensor respectively; the matrix product of the quantized first tensor and the quantized second tensor is used to determine the attention score tensor of the attention mechanism module.

[0032] In an exemplary embodiment, based on the above scheme, the large model quantization device further includes: a scaling scale determination module;

[0033] The scaling scale determination module includes a first determination unit, a second determination unit, and a third determination unit.

[0034] The first determining unit is used to determine the i-th first target value based on the value of the element in the i-th channel of the first tensor when there are outliers in the elements of at least one of the first tensor and the second tensor, where i is a positive integer not greater than the total number of channels of the first tensor; the second determining unit is used to determine the i-th second target value based on the value of the element in the i-th channel of the second tensor; and the third determining unit is used to determine the i-th scaling scale based on the i-th first target value, the i-th second target value, and the i-th scaling factor corresponding to the i-th channel; wherein the i-th scaling scale is used to scale the i-th channel of the first tensor and the i-th channel of the second tensor respectively.

[0035] In an exemplary embodiment, based on the above scheme, the first determining unit is specifically used to: determine the maximum value among the absolute values ​​of all elements in the i-th channel of the first tensor as the i-th first target value; or, determine the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​of all elements in the i-th channel of the first tensor as the i-th first target value.

[0036] In an exemplary embodiment, based on the above scheme, the third determining unit is specifically used to: determine an i-th first value based on the i-th first target value and the i-th scaling factor; determine an i-th second value based on the i-th second target value and the i-th scaling factor; and determine an i-th scaling scale based on the i-th first value and the i-th second value; wherein the i-th scaling scale is used to scale the i-th channel of the first tensor and the i-th channel of the second tensor.

[0037] In an exemplary embodiment, based on the above scheme, the third determining unit is further specifically used to: determine the ratio of the i-th first target value and the i-th second target value as the i-th scaling scale;

[0038] The scaling processing module described above is specifically used to: compare the element in the i-th channel of the first tensor with the i-th scaling scale to obtain the i-th channel in the scaled first tensor; and multiply the element in the i-th channel of the second tensor with the i-th scaling scale to obtain the i-th channel in the scaled second tensor.

[0039] In an exemplary embodiment, based on the above scheme, the large model quantization device further includes: a first scaling factor determination module;

[0040] The first scaling factor determination module is configured to: determine M candidate values ​​for the i-th scaling factor, where M is an integer greater than 1; determine a target candidate value for the i-th scaling scale based on the target candidate value of the i-th scaling factor, the i-th first target value, and the i-th second target value; obtain the predicted value of the i-th channel in the first tensor and the predicted value of the i-th channel in the second tensor under the target candidate value of the i-th scaling scale; determine target prediction information based on the predicted values ​​of the i-th channels in the first and second tensors obtained under the target candidate value of the i-th scaling scale; determine actual information based on the i-th channels in the first and second tensors; determine loss information based on the target prediction information and the actual information; and determine the candidate value corresponding to the minimum value among the M loss information as the i-th scaling factor.

[0041] In an exemplary embodiment, based on the above scheme, the large model quantization device further includes: a second scaling scale determination module;

[0042] The second scaling factor determination module is used to: determine an i-th first target value based on the value of the element in the i-th channel of the first tensor when there are outliers in at least one of the first tensor and the second tensor, where i is a positive integer not greater than the total number of channels of the first tensor; determine an i-th second target value based on the value of the element in the i-th channel of the second tensor; and determine an i-th scaling factor based on the i-th first target value, the i-th second target value, and the scaling factor; wherein the i-th scaling factor is used to scale the i-th channel of the first tensor and the second tensor respectively.

[0043] In an exemplary embodiment, based on the above scheme, the large model quantization device further includes: a second scaling factor determination module;

[0044] The second scaling factor determination module is configured to: determine M candidate values ​​for the scaling factor, where M is an integer greater than 1; determine a target candidate value for the i-th scaling scale based on the target candidate value of the scaling factor, the first target value corresponding to the first tensor, and the second target value corresponding to the second tensor; obtain a predicted first tensor corresponding to the first tensor and a predicted second tensor corresponding to the second tensor under the target candidate value of the i-th scaling scale; determine target prediction information based on the predicted first tensor and the predicted second tensor obtained under the target candidate value of the i-th scaling scale; determine actual information based on the first tensor and the second tensor; determine loss information based on the target prediction information and the actual information; and determine the candidate value corresponding to the minimum value among the M loss information as the scaling factor.

[0045] In an exemplary embodiment, based on the above scheme, the first tensor is a query tensor, and the second tensor is a key tensor;

[0046] The aforementioned quantization processing module is specifically used to: quantize the scaled first tensor and the scaled second tensor according to the per_token tag respectively to obtain the quantized first tensor and the quantized second tensor; or, quantize the scaled first tensor and the scaled second tensor according to the per_tensor tag respectively to obtain the quantized first tensor and the quantized second tensor.

[0047] In an exemplary embodiment, based on the above scheme, the first tensor is an attention score distribution tensor, and the second tensor is a value tensor, wherein the attention score distribution tensor is determined based on the attention score tensor;

[0048] The aforementioned quantization processing module is specifically used to: perform per_tensor quantization processing on the scaled first tensor to obtain the quantized first tensor; and perform per_token quantization processing on the scaled second tensor to obtain the quantized second tensor.

[0049] In an exemplary embodiment, based on the above scheme, the large model quantization device further includes: a smoothing quantization module;

[0050] The smoothing and quantization module is used to: determine the smoothing scale corresponding to the linear layer in the attention mechanism module; smooth the input and weights of the linear layer in the attention mechanism module according to the smoothing scale; and quantize the smoothed input and weights.

[0051] In an exemplary embodiment, based on the above scheme, the large model further includes a linear layer, and the smoothing and quantization module is further configured to: determine the smoothing scale corresponding to the j-th linear layer, where j is a positive integer; smooth the input and weights of the j-th linear layer according to the smoothing scale; and quantize the smoothed input and weights of the j-th linear layer; wherein the smoothing scales of the input projection and the gated projection in the attention mechanism module are equal.

[0052] In an exemplary embodiment, based on the above scheme, if the model structure before the linear layer is a normalized layer, the smoothing and quantization module is further configured to: smooth the weights of the normalized layer according to the smoothing scale of the current linear layer.

[0053] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the large model quantization method provided in the first aspect.

[0054] Fourthly, embodiments of this application provide a chip for implementing the large model quantization method provided in the first or second aspect above. Specifically, the chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to execute the large model quantization method provided in the first aspect above.

[0055] Fifthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to execute the large model quantization method provided in the first aspect.

[0056] Sixthly, embodiments of this application provide a computer program product, including computer program instructions that cause a computer to execute the large model quantization method provided in the first aspect.

[0057] In a seventh aspect, embodiments of this application provide a computer program that, when run on a computer, causes the computer to execute the large model quantization method provided in the first aspect.

[0058] In summary, the large model quantization scheme provided in this application involves the following steps for the quantization process of the attention mechanism module: First, the first and second tensors to be multiplied by matrix multiplication are scaled. In this application, scaling reduces the difference between outliers and other values ​​in the tensors, resulting in a more uniform tensor distribution and minimizing the impact of extreme values ​​on subsequent quantization parameters. Further, the scaled query tensor and scaled key tensor are then quantized separately. Since the first and second tensors have a more uniform tensor distribution after the scaling process, the limited quantization levels can be better utilized, thereby improving the accuracy of the quantized model. Therefore, the scheme provided in this application, by quantizing at least the activation values ​​of the attention mechanism module in the large model, can improve inference speed, while the quantized model exhibits higher inference accuracy. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the attention mechanism module;

[0061] Figure 2a and Figure 2b This is a schematic diagram of the quantification method;

[0062] Figure 3 A schematic diagram illustrating the quantization of the attention mechanism module for related technologies;

[0063] Figure 4a and Figure 4b This is a graph showing the numerical distribution of the tensor before and after scaling.

[0064] Figure 5 A flowchart illustrating the large model quantization method provided in this application embodiment;

[0065] Figure 6 A schematic diagram illustrating the distribution of query tensors, key tensors, or value tensors provided in embodiments of this application;

[0066] Figure 7 A schematic diagram showing the comparison of the numerical distribution of the query tensor and key tensor before and after scaling, provided in an embodiment of this application.

[0067] Figure 8 A flowchart illustrating the quantization process of the query tensor and key tensor in the attention mechanism module provided in this embodiment of the application;

[0068] Figure 9 This is a schematic diagram of the structure of the large model decoding layer;

[0069] Figure 10 A schematic block diagram of a large model quantization device provided in an embodiment of this application;

[0070] Figure 11 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein. In embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0073] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0074] Figure 1 This is a schematic diagram of the attention mechanism module.

[0075] refer to Figure 1 This diagram illustrates the basic process of the self-attention mechanism. Specifically:

[0076] Query tensors (Query_states, q_states), key tensors (Key_states, k_states), and value tensors (Value_states, v_states): In the attention mechanism, the input is first mapped to these three distinct spaces. The query, key, and value each correspond to three different linear transformations, typically performed using three different weight matrices. These transformations enable the inputs to interact and compute attention scores.

[0077] Transposing the key tensor k_states: In this step, k_states is transposed, which means that the shape of the tensor has changed, making it possible to perform matrix multiplication of k_states with q_states. Therefore, this transposition is to correctly align the dimensions of the attention computation so that k_states can be multiplied with q_states to obtain the attention score.

[0078] Batch matrix multiplication: Multiply q_states by the transpose of k_states. Batch matrix multiplication is a key step in calculating the attention score. It allows for similarity comparisons between queries and keys, and determines the importance of each position based on this similarity, resulting in an attention score tensor.

[0079] Softmax: The softmax function transforms attention scores into a probability distribution. It takes the attention score tensor from batch_matmul as input to the softmax function, which then converts the attention score tensor into a normalized probability distribution. Softmax ensures that the attention score at each position is within the interval [0,1], and that the sum of the scores at all positions equals 1. This step guarantees the interpretability and consistency of attention allocation.

[0080] Attention score distribution (p_states): This is the output of the softmax function, representing the attention score at each position. These scores describe the strength of the dependency between different positions in the input sequence.

[0081] Batch matrix multiplication: The final step is to perform batch matrix multiplication on p_states and v_states. This step applies the attention scores back to the values ​​to create the final output. Since p_states is a probability distribution, this step essentially means selecting the most important values ​​and combining them to form a new representation.

[0082] visible, Figure 1 The attention mechanism is illustrated, where queries, keys, and values ​​are used to calculate attention scores, which are then normalized using a softmax function. Finally, these scores are applied to the values ​​to produce a new representation. This process is crucial for understanding the relationships between different positions in an input sequence and has wide applications in tasks such as machine translation, text summarization, and question answering systems.

[0083] The purpose of quantizing large models is to reduce storage requirements, improve computational efficiency, and lower energy consumption, making them more suitable for deployment on resource-constrained devices such as mobile devices and embedded systems. Specifically, using lower-precision data types to represent weights and activation values ​​can significantly reduce model size. For example, converting from FP32 to INT8 can reduce the model size to one-quarter of its original size. Lower-precision computation is generally faster than higher-precision computation, especially on hardware accelerators that support specific low-precision operations. For instance, many modern graphics processors and dedicated AI chips are optimized for INT8 operations, thereby improving inference speed. Furthermore, lower-precision computation requires less energy, which is particularly important in mobile devices and other power-sensitive applications. Reduced energy consumption extends battery life and allows devices to operate more efficiently. Simultaneously, quantized models, due to their smaller size and lower computational requirements, are easier to deploy on edge devices. This is especially important for IoT applications, which often need to run models in resource-constrained environments.

[0084] Figure 2a and Figure 2b This is a schematic diagram of the quantification method.

[0085] refer to Figure 2a and Figure 2bThe diagram illustrates three different quantization methods: per-tensor quantization, per-token quantization, and per-channel quantization. These methods are all quantization compression techniques for neural network models, aiming to reduce computational and memory consumption, making them more suitable for operation in resource-constrained environments. Specifically:

[0086] Figure 2a The per-tensor quantization process is illustrated. In this method, the entire tensor (such as the weight matrix W or the input X) is quantized to a fixed quantization level. This means that all elements share the same scale factor (denoted by C) and use the same quantization step size. This method is simple to implement, but it does not fully utilize the diversity within the tensor because all elements are treated as having the same importance.

[0087] Figure 2b The left half illustrates the per-token quantization process. In this approach, each input token (e.g., each word or character in the input sequence) has its own scale. This means that the quantization level of each input token can be adjusted according to its own characteristics, rather than using a single scale on the entire input tensor. This method better reflects the characteristics of the input data, but it also increases computational complexity because each token needs to be processed individually.

[0088] Figure 2b The right half illustrates the per-channel quantization process. In this method, each output channel (e.g., each output channel of the convolution kernel) has its own scale. This means that the quantization level of each channel can be adjusted according to its own characteristics, rather than using a single scale on the entire output tensor. This method better reflects the characteristics of the output data, but it also increases computational complexity because each channel needs to be processed individually.

[0089] Figure 2b The lower half shows a combination of per_token and per_channel quantization. In this method, each input token and each output channel has its own scale. This is the finest quantization method, maximizing the use of data characteristics, but it also incurs the highest computational complexity.

[0090] The most effective large model compression scheme in the industry currently reduces model inference memory and accelerates computation by quantizing the weights and activations of linear layers. Building upon existing large model quantization schemes, this approach further explores the quantization possibilities of other modules, such as the attention mechanism. This will further improve inference speed while maintaining the accuracy of the quantized inference.

[0091] Figure 3 A schematic diagram illustrating the quantization of the attention mechanism module for related technologies.

[0092] refer to Figure 3 The illustrated related technology provides a quantization processing scheme for the attention mechanism module, specifically including the calculation process of forward Flash Attention (FMHA) under 8-bit quantization. Details are as follows:

[0093] 1. Quantized query and key matrix multiplication:

[0094]

[0095] Among them, Q INT8 and K INT8 These are quantized query and key matrices, which are 8-bit integers (INT8). Since the product of two 8-bit integers requires more bits to store, the product S... INT32 It is a 32-bit integer (INT32) matrix.

[0096] 2. Rescale the score matrix:

[0097]

[0098] Among them, S INT32 Convert from INT32 to floating-point (FP32) and apply an appropriate scaling factor to restore the original numerical range. d represents the dimension of the key matrix, α... q and α k These are the preset maximum absolute values, used to adjust the impact of quantization.

[0099] 3. Find the maximum value in each row:

[0100] m FP32 =rowmax(S FP32 )

[0101] 4. Construct a diagonal matrix:

[0102] M FP32 =diag(m FP32 )

[0103] 5. Calculate the softmax probability:

[0104] P FP32 =exp(S FP32 -M FP32 ·J)

[0105] The softmax probability is calculated by subtracting the maximum value and taking the exponent. Here, J is a matrix of all ones used for broadcast subtraction.

[0106] 6. Calculate the sum of probabilities for each row:

[0107] I FP32 =rowmax(P FP32 )

[0108] 7. Construct a diagonal matrix:

[0109] L FP32 =diag(I FP32 )

[0110] 8. Quantize the probability matrix to UINT8:

[0111]

[0112] 9. Performing matrix multiplication on the sum-valued matrix:

[0113] O INT32 =P UINT8 ·V INT8

[0114] 10. Rescale the output matrix:

[0115]

[0116] O INT32 Convert from INT32 back to floating-point and apply the appropriate scaling factor.

[0117] 11. Quantize the output matrix to INT8:

[0118]

[0119] Finally, O FP32 Quantize to the INT8 range to ensure all values ​​are between -127 and 127.

[0120] The quantization compression schemes provided by related technologies all use per-tensor methods for query tensors, key tensors, and value tensors. Users provide the quantization scaling factors (quant_scale) for Q, K, P, and V, which converts these tensors from FP32 to INT8 format. However, in actual business applications, when using the attention mechanism in INT8 format for inference, the error rate increases (e.g., the error rate reaches 10%), leading to decreased accuracy and failing to meet practical requirements.

[0121] The large model quantization scheme provided in this application can solve the accuracy degradation problem existing in related technologies. Specifically, in the processing of the attention mechanism module, the first and second tensors to be multiplied are first scaled. In this application embodiment, the scaling process can reduce the difference between outliers and other values ​​in the tensor, making the tensor distribution more uniform. Figure 4a The tensor shown is obtained after scaling as follows: Figure 4b The numerical distribution shown is relatively uniform. Further, the scaled query tensor and the scaled key tensor are quantized separately. Since the tensor distribution of the first and second tensors is more uniform after the above scaling process, the limited quantization levels can be better utilized, thereby improving the accuracy of the quantized model. It can be seen that the solution provided in this application, by quantizing at least the activation values ​​of the attention mechanism module in the large model, can improve inference speed, while the quantized model has higher inference accuracy.

[0122] The large model quantization method of this application will be described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0123] Figure 5 This is a flowchart illustrating the large model quantization method P500 provided in this embodiment of the application. Method P500 specifically quantizes the attention mechanism module in a large model. The execution entity of method P500 is a server or terminal.

[0124] When quantizing the query tensors (q_states) and key tensors (k_states) in the attention mechanism module, according to the torch.matmul(Q,KT) computation logic, the quantization methods include the following two cases: Case 1: Q (per_tensor) and K (per_tensor); Case 2: Q (per_token) + K (per_token). The reason why per_channel quantization is unsuitable is that both Q and K are two-dimensional matrices, with each element being an independent value, lacking a clear "channel" concept; the torch.matmul(Q,KT) matrix multiplication operation involves the elements of the entire matrix, not grouped by channel. Performing per_channel quantization on Q and K would complicate the computation logic and make it difficult to guarantee the correctness of the results. However, without using per_channel quantization, directly performing per_token or per_tensor quantization on the two tensors respectively presents challenges due to outliers in the channel dimension. Specifically, outliers cause uneven numerical distribution within the tensors, failing to fully utilize the limited quantization levels and negatively impacting the accuracy of the quantized model. Therefore, scaling is performed before performing the quantization process of the tensor described above.

[0125] In step S510, the first tensor and the second tensor in the attention mechanism module are scaled to obtain the scaled first tensor and the scaled second tensor.

[0126] For example, Figure 6 This is a schematic diagram illustrating the distribution of query tensors, key tensors, or value tensors provided in embodiments of this application. (Reference) Figure 6 As can be seen, the data of the tensors in the attention mechanism is not uniformly distributed. Specifically, the activation values ​​fluctuate significantly in some channels, with obvious peaks at the beginning and end of the channel dimension. To minimize the impact of extreme values ​​(outliers) on subsequent quantization parameters and thus better utilize the limited quantization levels to improve the accuracy of the quantized model, scaling is performed on the query tensor and key tensor in this embodiment.

[0127] In this embodiment, the first tensor and the second tensor are tensors used for matrix multiplication in the attention mechanism. For example, the following explanation uses the first tensor as the query tensor and the second tensor as the key tensor.

[0128] In the attention mechanism, if either the query tensor or the key tensor has outliers in its channel dimension, then to improve the accuracy of the quantized model, the channel dimension of the tensor needs to be scaled to optimize the distribution of element values ​​in that channel. It's important to note that when both the first and second tensors performing the matrix multiplication have outliers in their channel dimensions, the channels in which the outliers appear are not necessarily the same in the two tensors. For example, see [reference needed]. Figure 7 In the Q-distribution of the query tensor, outliers occur at channel identifier 1000, while in the K-distribution of the key tensor, outliers occur at channel identifier 2000. Therefore, in this embodiment, channel-by-channel scaling is performed on both tensors to optimize the distribution in each channel. Specifically, the i-th channel of the query tensor and the i-th channel of the key tensor are scaled using the i-th scaling scale. Before performing the scaling process, the method for determining the i-th scaling scale is explained. Specifically, the i-th scaling scale can be determined through steps S11-S13.

[0129] In step S11, the i-th first target value is determined based on the value of the element in the i-th channel of the first tensor, where i is a positive integer not greater than the total number of channels of the first tensor. And in step S12, the i-th second target value is determined based on the value of the element in the i-th channel of the second tensor.

[0130] In an exemplary embodiment, the maximum absolute value among all elements in the i-th channel of the first tensor can be used as the i-th first target value. If the element in the i-th channel of the query tensor is represented as q_states_i, then the i-th first target value can be represented as max(|q_states_i|).

[0131] Similarly, the maximum absolute value among all elements in the i-th channel of the second tensor can be taken as the i-th second objective value. If the elements in the i-th channel of the key tensor are represented as k_states_i, then the i-th second objective value can be represented as max(|k_states_i|).

[0132] In other exemplary embodiments, the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​in the i-th channel of the first tensor can also be determined as the i-th first target value. For example, the mean, median, etc., of the three elements with the largest absolute values ​​in the i-th channel of the query tensor can be used to obtain the i-th first target value.

[0133] Similarly, the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​in the i-th channel of the second tensor can be determined as the i-th first target value. For example, the mean, median, etc., of the three elements with the largest absolute values ​​in the i-th channel of the key tensor can be used to obtain the i-th second target value.

[0134] In step S13, the i-th scaling scale is determined based on the i-th first target value, the i-th second target value, and the scaling factor.

[0135] For example, the i-th scaling scale is determined by performing steps S131-S133.

[0136] In step S131, the i-th first value is determined based on the i-th first target value and the scaling factor α. In step S132, the i-th second value is determined based on the i-th second target value and the scaling factor α. And, in step S133, the i-th scaling scale is determined based on the i-th first value and the i-th second value.

[0137] For example, taking max(|q_states_i|) and max(|k_states_i|) as examples, the i-th first target value and the i-th second target value are the same as the above-mentioned zoom_scale. i It can be expressed as the ratio of the i-th first value to the i-th second value, as shown in formula (1).

[0138]

[0139] Where, [max(|q_states_i|)] α This represents the i-th first objective value mentioned above, [max(|k_states_i|)]. 1-α This represents the i-th second objective value mentioned above.

[0140] In other embodiments, the i-th scaling scale mentioned above is zoom_scale i It can also be expressed as the ratio of the i-th second value to the i-th first value, as shown in formula (2).

[0141]

[0142] To ensure consistency between the matrix product of the scaled first and second tensors and the matrix product of the unscaled first and second tensors, the two tensors are magnified and shrunk respectively during the scaling process. Specifically, `zoom_scale`... i The scaling methods for the two tensors differ depending on the representation method described above.

[0143] When the above formula (1) is used to represent the i-th scaling scale, the scaling processing of the i-th channel of the first tensor and the i-th channel of the second tensor is as follows: steps S510-1 and S510-2.

[0144] In step S510-1, the element in the i-th channel of the first tensor is compared with the i-th scaling scale to obtain the i-th channel in the scaled first tensor.

[0145] For example, the query tensor is used to map all elements q_states_i in the i-th channel to the aforementioned i-th scaling scale zoom_scale. i In contrast, the i-th channel q_states_i′ in the scaled query tensor is obtained, as shown in formula (3).

[0146]

[0147] In step S510-2, the element in the i-th channel of the second tensor is multiplied by the i-th scaling factor to obtain the i-th channel in the scaled second tensor.

[0148] For example, all elements k_states_i in the i-th channel of the key tensor are respectively compared with the i-th scaling scale zoom_scale. i Multiplying them together, we obtain the i-th channel k_states_i′ in the scaled key tensor, as shown in formula (4).

[0149] k_states_i′=k_states_i*zoom_scale i (4)

[0150] When the above formula (2) is used to represent the i-th scaling scale, the scaling processing of the i-th channel of the first tensor and the i-th channel of the second tensor is as follows: steps S510-1' and S510-2'.

[0151] In step S510-1', the element in the i-th channel of the first tensor is multiplied by the i-th scaling factor to obtain the i-th channel in the scaled first tensor.

[0152] For example, the query tensor is used to map all elements q_states_i in the i-th channel to the aforementioned i-th scaling scale zoom_scale. i Multiplying them together, we get the i-th channel q_states_i′ in the scaled query tensor, as shown in formula (5).

[0153] q_states_i′=q_states_i*zoom_scale i (5)

[0154] In step S510-2', the elements in the i-th channel of the second tensor are compared with the i-th scaling scale to obtain the i-th channel in the scaled second tensor.

[0155] For example, all elements k_states_i in the i-th channel of the key tensor are respectively compared with the i-th scaling scale zoom_scale. i In contrast, the i-th channel k_states_i′ in the scaled key tensor is obtained, as shown in formula (6).

[0156]

[0157] In some embodiments, to improve scaling efficiency, the scaling factor can be the same for each channel of the first and second tensors. In other embodiments, to improve scaling accuracy and obtain tensors with more uniform numerical distribution, different scaling factors can be set for different channels, i.e., the i-th scaling factor α corresponding to the i-th channel. i Specifically, the scaling factor in formulas (1) and (2) above can be replaced with α. i .

[0158] In an exemplary embodiment, the scaling factor described above may be a given value.

[0159] In an exemplary embodiment, to improve the efficiency and accuracy of large model quantization processing, the scaling factor can be determined through an automated search. As mentioned above, the scaling factor falls into two categories: one where the scaling factor is used for scaling each channel of the entire tensor, and the other where the i-th scaling factor α... i Used for scaling operations on the i-th channel of a tensor.

[0160] First, let's introduce the i-th scaling factor α. i The automated determination of the implementation example.

[0161] Specifically, the i-th scaling factor α can be determined automatically through steps S21-S27. i The i-th scaling factor α i This will be used for scaling the i-th channel in the first and second tensors.

[0162] In step S21, the i-th scaling factor α is determined. i There are M candidate values, where M is an integer greater than 1.

[0163] For example, the i-th scaling factor α i The candidate value set is [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9].

[0164] In step S22, the target candidate value of the i-th scaling scale is determined based on the target candidate value (e.g., 0.2) in the i-th scaling factor, the i-th first target value, and the i-th second target value.

[0165] If the candidate value of the i-th scaling factor is 0.2, the candidate value of the scaling factor 0.2, the i-th first target value, and the i-th second target value can be input into formula (7) or formula (8) to determine the candidate value of the i-th scaling scale.

[0166]

[0167] In step S23, the predicted value of the i-th channel in the first tensor and the predicted value of the i-th channel in the second tensor are obtained under the target candidate value at the i-th scaling scale.

[0168] The i-th channel in the first tensor and the second tensor are scaled using the target candidate value at the i-th scaling scale, respectively. Then, the i-th channel in the first tensor and the second tensor (represented as q_states_i′ and k_states_i′) after scaling are quantized, respectively. This allows us to determine the predicted value q_states_i″ of the i-th channel in the first tensor and the predicted value k_states_i″ of the i-th channel in the second tensor.

[0169] In step S24, target prediction information is determined based on the predicted values ​​of the i-th channels in the first tensor and the second tensor obtained under the target candidate value at the i-th scaling scale. And in step S25, actual information is determined based on the i-th channel in the first tensor and the i-th channel in the second tensor.

[0170] In one exemplary embodiment, on one hand, the target prediction information is obtained by multiplying the predicted value q_states_i″ of the i-th channel in the query tensor and the transpose of the predicted value k_states_i″ of the i-th channel in the key tensor. On the other hand, the actual information P_i is obtained by multiplying the original value q_states_i of the i-th channel in the query tensor before scaling and the transpose of the original value k_states_i of the i-th channel in the key tensor before scaling.

[0171] In another exemplary embodiment, on one hand, the predicted value q_states_i″ of the i-th channel in the query tensor and the predicted value k_states_i″ of the i-th channel in the key tensor are calculated, and the statistical value between the two vectors is used to obtain the target prediction information. On the other hand, the original values ​​q_states_i of the i-th channel in the query tensor before scaling and k_states_i of the i-th channel in the key tensor before scaling are calculated. The statistical value between the two vectors is used to obtain the actual information P_i. The statistical value mentioned above can be the mean, sum, sum of squares, etc. of the absolute values ​​of the elements in the two vectors.

[0172] In step S26, loss information is determined based on the target prediction information and the actual information.

[0173]

[0174] The aforementioned loss information Loss_i represents the degree of difference in the vector product before and after scaling and quantizing the i-th channel of the two tensors.

[0175] In step S27, the candidate value corresponding to the minimum value among the M loss information is determined as the i-th scaling factor α. i .

[0176] The smaller the loss information Loss_i, the smaller the difference in the product between the i-th channel of the two tensors before and after scaling and quantization. Therefore, the candidate value corresponding to the minimum value among the M loss information is used as the i-th scaling factor α. i This helps to reduce the impact of scaling on the matrix to a greater extent and helps to ensure the inference accuracy of the model.

[0177] Next, we will introduce an example of automatically determining the global scaling factor α.

[0178] Specifically, the global scaling factor α can be automatically determined through steps S31-S37, and this global scaling factor α will be used for scaling processing of each channel of the first tensor and the second tensor.

[0179] In step S31, M candidate values ​​for the scaling factor are determined, where M is an integer greater than 1.

[0180] For example, the candidate set of global scaling factor α is [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9].

[0181] In step S32, the target candidate value of the i-th scaling scale is determined based on the target candidate value of the scaling factor (e.g., 0.2), the first target value corresponding to the first tensor, and the second target value corresponding to the second tensor.

[0182] For example, the first target value corresponding to the first tensor can be the maximum absolute value of the elements in the first tensor, max(|q_states|). Similarly, the second target value corresponding to the second tensor can be the maximum absolute value of the elements in the second tensor, max(|k_states|).

[0183] For example, the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​in the first tensor can be determined as the first target value corresponding to the first tensor. For instance, the mean, median, etc., of the five elements with the largest absolute values ​​in the first tensor of the query tensor can be used to calculate the first target value corresponding to the first tensor. Similarly, the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​in the second tensor can be determined as the second target value corresponding to the second tensor. For instance, the mean, median, etc., of the five elements with the largest absolute values ​​in the second tensor of the query tensor can be used to calculate the second target value corresponding to the second tensor.

[0184] Furthermore, if the target candidate value of the scaling factor is 0.2, the candidate value of the scaling factor 0.2, the first target value corresponding to the first tensor, and the second target value corresponding to the second tensor can be input into formula (9) or formula (10) to determine the candidate value of the scaling scale.

[0185]

[0186] In step S33, the predicted first tensor corresponding to the first tensor and the predicted second tensor corresponding to the second tensor are obtained under the target candidate value at the i-th scaling scale.

[0187] If both the first and second tensors contain X channels, then first calculate X scaling scales according to formulas (9) and (10), and then scale the X channels in the tensor using M scaling scales respectively. For example, scale the X channels in the query tensor using M scaling scales to obtain the scaled query tensor (represented as q_states′); scale the X channels in the key tensor using M scaling scales to obtain the scaled key tensor (represented as k_states′). Further, quantize the scaled first tensor to obtain the predicted first tensor; for example, quantize the scaled query tensor q_states′ to obtain the predicted query tensor q_states″. Similarly, quantize the scaled second tensor to obtain the predicted key tensor; for example, quantize the scaled key tensor k_states′ to obtain the predicted key tensor k_states″.

[0188] In step S34, target prediction information is determined based on the first prediction tensor and the second prediction tensor obtained under the target candidate value at the i-th scaling scale. And in step S35, actual information is determined based on the first tensor and the second tensor.

[0189] In one exemplary embodiment, on one hand, the target prediction information is obtained by multiplying the transposes of the prediction query tensor q_states″ and the prediction key tensor k_states″. On the other hand, multiplying the transpose of the original query tensor q_states and the original key tensor k_states yields the actual information P.

[0190] In another exemplary embodiment, on one hand, the statistical values ​​between the prediction query tensor q_states″ and the prediction key tensor k_states″ are calculated to obtain the target prediction information. On the other hand, the actual information P is obtained by taking the statistical values ​​between the original query tensor q_states and the original key tensor k_states. These statistical values ​​can be the mean, sum, or sum of squares of the absolute values ​​of the elements in the two tensors.

[0191] In step S36, loss information is determined based on the target prediction information and the actual information;

[0192] Loss = loss_func(P′) α＝0.2 P)

[0193] The aforementioned loss information represents the degree of difference in the matrix product before and after scaling and quantization of the two tensors.

[0194] In step S37, the candidate value corresponding to the minimum value among the M loss information is determined as the scaling factor.

[0195] The smaller the loss value mentioned above, the smaller the matrix product is before and after scaling and quantization of the two tensors. Therefore, using the candidate value corresponding to the minimum value among the M loss values ​​as the global scaling factor α helps to reduce the impact of scaling on the matrix and ensures the inference accuracy of the model.

[0196] Understandably, depending on the actual needs of model quantization (such as accuracy and efficiency), when scaling the i-th channel in the first and second tensors, one can choose the aforementioned global scaling factor, or choose the i-th scaling factor α corresponding to the i-th channel. i As can be seen, the embodiments of this application not only provide an automated search method with excellent scaling, but also improve the flexibility of scaling activation values ​​in the attention mechanism module, which is beneficial to expanding the application scenarios and scope.

[0197] In an exemplary embodiment, Figure 7 This is a comparative diagram showing the scaling of the i-th channel of the first and second tensors before and after scaling, as provided in an embodiment of this application. (Reference) Figure 7 As can be seen, for the Q-distribution of the query tensor pairs, outliers appear at channel 1000, which can be addressed by zoom_scale. i＝1000 Scaling this channel can reduce the anomaly of outliers and optimize the Q-distribution to make its values ​​more evenly distributed, thus benefiting subsequent quantization. Similarly, refer to... Figure 7 As can be seen, for a K-distribution of tensor pairs, outliers occur at channel 2000, which can be addressed by zoom_scale. i＝2000 Scaling the channel can reduce the anomalousness of outliers and optimize the K distribution to make the values ​​more evenly distributed, which is beneficial for subsequent quantization.

[0198] In an exemplary embodiment, if per-channel quantization is not possible for the value tensor, per-token or per-tensor quantization must be performed. As mentioned earlier, the value tensor is unevenly distributed, and outliers consistently appear in certain channel dimensions. Therefore, to improve quantization accuracy, the value tensor also needs to be scaled. Specifically, to ensure consistency with the product of the p_states matrix before and after scaling, both p_states and the value tensor need to be scaled synchronously. The specific scaling method is similar to the processing of the query tensor and key tensor in the above embodiments, and will not be repeated here.

[0199] Continue to refer to Figure 5 In step S520, the scaled first tensor and the scaled second tensor are quantized respectively to obtain the quantized first tensor and the quantized second tensor; wherein, the matrix product of the quantized first tensor and the quantized second tensor is used to determine the attention score tensor of the attention mechanism module.

[0200] In an exemplary embodiment, after the above scaling process, a first tensor and a second tensor with a more uniform numerical distribution are obtained. Next, the two scaled tensors are quantized respectively.

[0201] Exemplary Reference Figure 8 Given that the first tensor and the second tensor are the query tensor (q_states) and the key tensor (k_states) respectively, after zoom_scale is applied to each channel of the two tensors (the specific implementation is as described in the embodiment corresponding to S510, which will not be repeated here), the scaled query tensor (q_states) and the scaled key tensor (k_states) are then quantized respectively.

[0202] As mentioned before, according to the calculation logic of torch.matmul(Q,KT), there are two ways to quantize it: Case 1: Q(per_tensor) and K(per_tensor); Case 2: Q(per_token)+K(per_token).

[0203] refer to Figure 8 After quantizing the scaled query tensor (q_states) and the scaled key tensor (k_states), the quantized key tensor can be transposed and then batch matrix multiplication can be performed to obtain the attention scores tensor (attention scores_states).

[0204] Understandably, reference Figure 1 The attention scores tensor (attention scores_states) is then subjected to softmax to obtain the attention score distribution tensor (p_states). Further, the attention score distribution tensor (p_states) is batch-multiplied with the value tensor (v_states) to finally obtain the output information of the attention mechanism module.

[0205] In an exemplary embodiment, the first and second tensors described above can also be attention score distributions (p_states) and value tensors (v_states). In this case, since p_states are generated by softmax, the values ​​in p_states are all between 0 and 1, meaning the element values ​​within p_states are evenly distributed. Therefore, since the value tensor does not require scaling, p_states also does not need to be scaled. Furthermore, since p_states is a one-dimensional vector, per-tensor quantization can be used. The value tensor can be quantized per-tensor, per-token, or per-channel. Also, since outliers in the value tensor generally appear in the channel dimension, per-channel quantization can be used, which is beneficial for the quantization efficiency of large models.

[0206] Understandably, if per-channel quantization is not possible for value tensors, scaling operations must be performed on p_states and value tensors before per-token or per-tensor quantization.

[0207] In the quantization scheme for the attention mechanism module in the large model provided by Method P500, the first and second tensors to be multiplied by matrix multiplication are first scaled. In this embodiment, scaling reduces the difference between outliers and other values ​​in the tensors, making the tensor distribution more uniform and minimizing the impact of extreme values ​​on subsequent quantization parameters. Furthermore, the scaled query tensor and scaled key tensor are quantized separately. Since the first and second tensors have a more uniform tensor distribution after the scaling process, the limited quantization levels can be better utilized, thereby improving the accuracy of the quantized model. Therefore, the scheme provided in this embodiment, by quantizing at least the activation values ​​of the attention mechanism module in the large model, can improve inference speed, while the quantized model exhibits higher inference accuracy.

[0208] The above embodiments described the quantization processing of activation values ​​(including query tensors, key tensors, p_states, and value tensors) in the attention mechanism. In other embodiments, the linear transformation layer in the attention mechanism can also be quantized. Specifically, steps S41-S43 can be executed.

[0209] In step S41, the smoothing scale corresponding to the linear transformation layer in the attention mechanism module is determined.

[0210]

[0211] Where X represents the input, W represents the weight, max() represents the maximum value, β represents the smoothing factor, and j represents the number of channels.

[0212] In some embodiments, β can be a fixed value.

[0213] In other embodiments, β can also be an optimal value determined through an automated search. The steps are as follows:

[0214] 1) Use calibration data to statistically analyze the input scale and weight scale of the linear layer;

[0215] 2) Design the loss function loss_func

[0216] Forβin[0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]

[0217] Calculate smooth_scale according to formula (11)

[0218] For input in examples:

[0219] Input_edit=input.div(smooth_scale)

[0220] Weight_edit=weight.mul(smooth_scale)

[0221] Y_hat=F.Linear(Quant(input_edit),Quant(weight_edit))

[0222] Y = F.linear(input, weight)

[0223] Loss = loss_func(Y_hat, Y)

[0224] Choose the β value corresponding to the minimum loss.

[0225] In step S42, the inputs and weights of the linear layer in the attention mechanism module are smoothed according to the smoothing scale.

[0226] For example, the input of a smoothed linear layer is represented as: input = input.div(smooth_scale);

[0227] The weights of the smoothed linear layer are represented as: weight = weight.mul(smooth_scale).

[0228] In step S43, the smoothed input and weights are quantized.

[0229] The output after linear layer quantization is represented as: output = F.linear(Quant(input)).Quant(weight).

[0230] The embodiments of this application reduce quantization errors by using smoothing processing, so that the quantized data is as close as possible to the original data, thereby improving quantization accuracy.

[0231] In an exemplary embodiment, reference is made to Figure 9 The large model decoding layer shown includes a normalization layer (LayerNorm), a linear layer (W_pack), an attention mechanism module, a linear layer projection, a linear layer (Up_proj), a linear layer (Gate_proj), and a linear layer (Down_proj).

[0232] The large model quantization scheme provided in this application also includes quantization processing for any one of the linear layers shown in the figure (the j-th linear layer). Specifically, steps S51-S53 can be executed.

[0233] In step S51, the smoothing scale corresponding to the j-th linear layer is determined, where j is a positive integer.

[0234] For specific implementation details, please refer to the embodiment corresponding to step S41, which will not be repeated here.

[0235] In step S52, the input and weights of the j-th linear layer are smoothed according to the smoothing scale.

[0236] For specific implementation details, please refer to the embodiment corresponding to step S42, which will not be repeated here.

[0237] In step S53, the input and weights of the j-th linear layer after smoothing are quantized.

[0238] For specific implementation details, please refer to the embodiment corresponding to step S43, which will not be repeated here.

[0239] It should be noted that if the model structure preceding the linear layer is a normalized layer, this embodiment further processes the weights of the normalized layer according to the smoothing scale corresponding to the current linear layer. Specifically, Layernorm weightln.weight = ln.weight.div(smooth_scale). By performing appropriate smoothing processing on the weights of the LayerNorm layer, it can be ensured that the model's behavior remains consistent when smoothing the input data. This method is particularly useful in quantization and other processing, reducing performance loss caused by smoothing.

[0240] For example, refer to Figure 9 Since the linear layer input projection (up_proj) and gated projection (gate_proj) in the large model shown in the figure share a single input, when designing the smoothing mechanism for the linear layer up_proj and gate_proj, the smoothing scale of the linear layer up_proj and gate_proj can be set to the same value to improve the smoothing effect.

[0241] By using the smooth quantization method described above for large models, the accuracy of the quantized large models is higher than that of the open-source smoothquant solution provided by related technologies. At the same time, smooth quantization can significantly reduce the demand for computing resources, thereby improving the inference speed of the model and achieving a higher speedup ratio.

[0242] In the large model quantization scheme provided in this application embodiment, the activation values ​​in the attention mechanism module are scaled before quantization. Furthermore, the aforementioned smoothing process can be performed on the weights and inputs of the linear layers in the large model before quantization. This can further improve the inference speed of the quantized large model. For example, compared to the embodiment that only performs the aforementioned smoothing process on the linear layers in the large model, the embodiment that further performs scaling and quantization of the activation values ​​in the attention mechanism module as provided by method P500 accelerates the inference of the large model by approximately 10%.

[0243] As can be seen, the large model quantization scheme provided in this application is a Post-Training Quantization (PTQ) method, which requires no training and finds the optimal hyperparameters automatically, ensuring accuracy after quantization. The large model quantization method provided in this scheme can be integrated into a toolkit (tilearn.llm), supporting automated quantization of large models. Specifically, it can be used for large models in any language that require inference acceleration, making it widely applicable.

[0244] The above text combined Figures 1 to 9 The method embodiments of this application are described below, in conjunction with Figure 10Embodiments of the large-scale model quantization device described in this application, and in conjunction with Figure 10 An embodiment of the large model quantization device of this application is described.

[0245] Figure 10 This is a schematic block diagram of a large model quantization device 1000 provided in an embodiment of this application.

[0246] refer to Figure 10 The large model quantization device 1000 provided in this application embodiment includes: a scaling processing module 1010 and a quantization processing module 1020.

[0247] The scaling module 1010 is used to scale the first and second tensors in the large model, including the attention mechanism module, to obtain scaled first and scaled second tensors, respectively. The quantization module 1020 is used to quantize the scaled first and scaled second tensors to obtain quantized first and quantized second tensors, respectively. The matrix product of the quantized first and quantized second tensors is used to determine the attention score tensor of the attention mechanism module.

[0248] In an exemplary embodiment, based on the above scheme, the large model quantization device 1000 further includes: a first scaling scale determination module;

[0249] The aforementioned first scaling scale determination module includes a first determination unit, a second determination unit, and a third determination unit;

[0250] The first determining unit is used to determine the i-th first target value based on the value of the element in the i-th channel of the first tensor when there are outliers in the elements of at least one of the first tensor and the second tensor, where i is a positive integer not greater than the total number of channels of the first tensor; the second determining unit is used to determine the i-th second target value based on the value of the element in the i-th channel of the second tensor; and the third determining unit is used to determine the i-th scaling scale based on the i-th first target value, the i-th second target value, and the i-th scaling factor corresponding to the i-th channel; wherein the i-th scaling scale is used to scale the i-th channel of the first tensor and the i-th channel of the second tensor respectively.

[0251] In an exemplary embodiment, based on the above scheme, the first determining unit is specifically used to: determine the maximum value among the absolute values ​​of all elements in the i-th channel of the first tensor as the i-th first target value; or, determine the statistical value of the absolute values ​​of the N elements with the largest absolute values ​​among all elements in the i-th channel of the first tensor as the i-th first target value.

[0252] In an exemplary embodiment, based on the above scheme, the third determining unit is specifically used to: determine an i-th first value based on the i-th first target value and the i-th scaling factor; determine an i-th second value based on the i-th second target value and the i-th scaling factor; and determine an i-th scaling scale based on the i-th first value and the i-th second value; wherein the i-th scaling scale is used to scale the i-th channel of the first tensor and the i-th channel of the second tensor.

[0253] In an exemplary embodiment, based on the above scheme, the third determining unit is further specifically used to: determine the ratio of the i-th first target value and the i-th second target value as the i-th scaling scale;

[0254] The scaling processing module 1010 is specifically used to: compare the element in the i-th channel of the first tensor with the i-th scaling scale to obtain the i-th channel in the scaled first tensor; and multiply the element in the i-th channel of the second tensor with the i-th scaling scale to obtain the i-th channel in the scaled second tensor.

[0255] In an exemplary embodiment, based on the above scheme, the large model quantization device 1000 further includes: a first scaling factor determination module;

[0256] The first scaling factor determination module is configured to: determine M candidate values ​​for the i-th scaling factor, where M is an integer greater than 1; determine a target candidate value for the i-th scaling scale based on the target candidate value of the i-th scaling factor, the i-th first target value, and the i-th second target value; obtain the predicted value of the i-th channel in the first tensor and the predicted value of the i-th channel in the second tensor under the target candidate value of the i-th scaling scale; determine target prediction information based on the predicted values ​​of the i-th channels in the first and second tensors obtained under the target candidate value of the i-th scaling scale; determine actual information based on the i-th channels in the first and second tensors; determine loss information based on the target prediction information and the actual information; and determine the candidate value corresponding to the minimum value among the M loss information as the i-th scaling factor.

[0257] In an exemplary embodiment, based on the above scheme, the large model quantization device 1000 further includes: a second scaling scale determination module;

[0258] The second scaling factor determination module is used to: determine an i-th first target value based on the value of the element in the i-th channel of the first tensor when there are outliers in at least one of the first tensor and the second tensor, where i is a positive integer not greater than the total number of channels of the first tensor; determine an i-th second target value based on the value of the element in the i-th channel of the second tensor; and determine an i-th scaling factor based on the i-th first target value, the i-th second target value, and the scaling factor; wherein the i-th scaling factor is used to scale the i-th channel of the first tensor and the second tensor respectively.

[0259] In an exemplary embodiment, based on the above scheme, the large model quantization device 1000 further includes: a second scaling factor determination module;

[0260] The second scaling factor determination module is configured to: determine M candidate values ​​for the scaling factor, where M is an integer greater than 1; determine a target candidate value for the i-th scaling scale based on the target candidate value of the scaling factor, the first target value corresponding to the first tensor, and the second target value corresponding to the second tensor; obtain a predicted first tensor corresponding to the first tensor and a predicted second tensor corresponding to the second tensor under the target candidate value of the i-th scaling scale; determine target prediction information based on the predicted first tensor and the predicted second tensor obtained under the target candidate value of the i-th scaling scale; determine actual information based on the first tensor and the second tensor; determine loss information based on the target prediction information and the actual information; and determine the candidate value corresponding to the minimum value among the M loss information as the scaling factor.

[0261] In an exemplary embodiment, based on the above scheme, the first tensor is a query tensor, and the second tensor is a key tensor;

[0262] The aforementioned quantization processing module 1020 is specifically used to: quantize the aforementioned scaled first tensor and the aforementioned scaled second tensor according to the tag per_token, respectively, to obtain the quantized first tensor and the quantized second tensor; or, quantize the aforementioned scaled first tensor and the aforementioned scaled second tensor according to the tensor per_tensor, respectively, to obtain the quantized first tensor and the quantized second tensor.

[0263] In an exemplary embodiment, based on the above scheme, the first tensor is an attention score distribution tensor, and the second tensor is a value tensor, wherein the attention score distribution tensor is determined based on the attention score tensor;

[0264] The aforementioned quantization processing module 1020 is specifically used for: performing per_tensor quantization processing on the aforementioned scaled first tensor to obtain the quantized first tensor; and performing per_token quantization processing on the aforementioned scaled second tensor to obtain the quantized second tensor.

[0265] In an exemplary embodiment, based on the above scheme, the large model quantization device 1000 further includes: a smoothing quantization module;

[0266] The smoothing and quantization module is used to: determine the smoothing scale corresponding to the linear layer in the attention mechanism module; smooth the input and weights of the linear layer in the attention mechanism module according to the smoothing scale; and quantize the smoothed input and weights.

[0267] In an exemplary embodiment, based on the above scheme, the large model further includes a linear layer, and the smoothing and quantization module is further configured to: determine the smoothing scale corresponding to the j-th linear layer, where j is a positive integer; smooth the input and weights of the j-th linear layer according to the smoothing scale; and quantize the smoothed input and weights of the j-th linear layer; wherein the smoothing scales of the input projection and the gated projection in the attention mechanism module are equal.

[0268] In an exemplary embodiment, based on the above scheme, if the model structure before the linear layer is a normalized layer, the smoothing and quantization module is further configured to: smooth the weights of the normalized layer according to the smoothing scale of the current linear layer.

[0269] It should be understood that, as Figure 10 The embodiment of the large model quantization device shown corresponds to the embodiment of the large model quantization method described above, and a similar description can be found in the method embodiment. To avoid repetition, it will not be repeated here. Specifically, through... Figure 10 The information interaction between the various modules in the large model quantization device shown can execute the above-described embodiment of the large model quantization method, through methods such as... Figure 10 The information interaction between the modules in the large model quantization device shown can execute the above-described embodiments of the large model quantization method. For the sake of brevity, the method embodiments corresponding to the aforementioned and other operations and / or functions of each module in the device will not be described again here.

[0270] The above description, in conjunction with the accompanying drawings, describes the operation and maintenance related apparatus of the software agent according to the embodiments of this application from the perspective of functional modules. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0271] This application also provides an electronic device.

[0272] Figure 11 This is a schematic block diagram of an electronic device 1100 provided in an embodiment of this application. As described above, the operation and maintenance related devices of the software agent can be deployed in, for example... Figure 11 The electronic device shown can therefore be used to perform the large model quantization method described above.

[0273] like Figure 11 As shown, the electronic device 1100 may include:

[0274] The system includes a memory 1110 and a processor 1120. The memory 1110 stores a computer program 1130 and transfers the program code 1130 to the processor 1120. In other words, the processor 1120 can call and run the computer program 1130 from the memory 1110 to implement the methods in the embodiments of this application.

[0275] For example, the processor 1120 can be used to execute the steps in the large model quantization method described above, or to execute the steps in the large model quantization method described above, according to the instructions in the computer program 1130.

[0276] In some embodiments of this application, the processor 1120 may include, but is not limited to:

[0277] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0278] In some embodiments of this application, the memory 1110 includes, but is not limited to:

[0279] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0280] In some embodiments of this application, the computer program 1130 may be divided into one or more modules, which are stored in the memory 1110 and executed by the processor 1120 to complete the large model quantization method provided in this application, or to complete the steps in the large model quantization method described above. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 1130 in the electronic device.

[0281] like Figure 11As shown, the electronic device 1100 may further include:

[0282] Transceiver 1140, which can be connected to processor 1120 or memory 1110.

[0283] The processor 1120 can control the transceiver 1140 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 1140 may include a transmitter and a receiver. The transceiver 1140 may further include antennas, and the number of antennas may be one or more.

[0284] It should be understood that the various components in the electronic device 1100 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0285] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0286] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0287] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Video Disc (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)).

[0288] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0289] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0290] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0291] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for quantizing large models, characterized in that, The large model includes an attention mechanism module, and the method includes: The first and second tensors in the attention mechanism module are scaled to obtain scaled first and scaled second tensors respectively. The scaled first tensor and the scaled second tensor are quantized respectively to obtain the quantized first tensor and the quantized second tensor. The matrix product of the first quantized tensor and the second quantized tensor is used to determine the attention score tensor of the attention mechanism module.

2. The method according to claim 1, characterized in that, If outliers exist in the elements of at least one of the first tensor and the second tensor in the channel dimension, the method further includes: Based on the value of the element in the i-th channel of the first tensor, determine the i-th first target value, where i is a positive integer not greater than the total number of channels of the first tensor; The i-th second target value is determined based on the value of the element in the i-th channel of the second tensor; The i-th scaling scale is determined based on the i-th first target value, the i-th second target value, and the i-th scaling factor corresponding to the i-th channel; The i-th scaling scale is used to scale the i-th channel in the first tensor and the second tensor respectively.

3. The method according to claim 2, characterized in that, Determining the i-th first target value based on the value of the element in the i-th channel of the first tensor includes: The maximum absolute value among all elements in the i-th channel of the first tensor is determined as the i-th first target value; or, The statistical value of the absolute values ​​of the N elements with the largest absolute values ​​in the i-th channel of the first tensor is determined as the i-th first target value.

4. The method according to claim 2, characterized in that, The step of determining the i-th scaling scale based on the i-th first target value, the i-th second target value, and the i-th scaling factor corresponding to the i-th channel includes: The i-th first value is determined based on the i-th first target value and the i-th scaling factor; The i-th second value is determined based on the i-th second target value and the i-th scaling factor; The i-th scaling factor is determined based on the i-th first value and the i-th second value; The i-th scaling scale is used to scale the i-th channel of the first tensor and the i-th channel of the second tensor.

5. The method according to claim 4, characterized in that, Determining the i-th scaling scale based on the i-th first value and the i-th second value includes: The ratio of the i-th first value to the i-th second value is determined as the i-th scaling factor; The scaling process performed on the first and second tensors in the attention mechanism module to obtain scaled first and scaled second tensors includes: The element in the i-th channel of the first tensor is compared with the i-th scaling scale to obtain the i-th channel in the scaled first tensor. Multiply the element in the i-th channel of the second tensor by the i-th scaling factor to obtain the i-th channel in the scaled second tensor.

6. The method according to claim 2, characterized in that, The method further includes: Determine M candidate values ​​for the i-th scaling factor, where M is an integer greater than 1; Based on the target candidate value of the i-th scaling factor, the i-th first target value, and the i-th second target value, determine the target candidate value of the i-th scaling scale; Obtain the predicted value of the i-th channel in the first tensor and the predicted value of the i-th channel in the second tensor at the target candidate value of the i-th scaling scale. Target prediction information is determined based on the predicted values ​​of the first and second tensors in the first and second tensors obtained under the target candidate values ​​at the i-th scaling scale. The actual information is determined based on the i-th channel in the first tensor and the i-th channel in the second tensor; Loss information is determined based on the target prediction information and the actual information; The candidate value corresponding to the minimum value among the M loss information is determined as the i-th scaling factor.

7. The method according to claim 1, characterized in that, If outliers exist in the elements of at least one of the first tensor and the second tensor in the channel dimension, the method further includes: Based on the value of the element in the i-th channel of the first tensor, determine the i-th first target value, where i is a positive integer not greater than the total number of channels of the first tensor; The i-th second target value is determined based on the value of the element in the i-th channel of the second tensor; The i-th scaling factor is determined based on the i-th first target value, the i-th second target value, and the scaling factor. The i-th scaling scale is used to scale the i-th channel in the first tensor and the second tensor respectively.

8. The method according to claim 7, characterized in that, The method further includes: Determine M candidate values ​​for the scaling factor, where M is an integer greater than 1; Based on the target candidate values ​​of the scaling factor, the first target value corresponding to the first tensor, and the second target value corresponding to the second tensor, determine the target candidate value of the i-th scaling scale; Obtain the predicted first tensor corresponding to the first tensor and the predicted second tensor corresponding to the second tensor under the target candidate value at the i-th scaling scale; Target prediction information is determined based on the first and second predicted tensors obtained at the target candidate values ​​at the i-th scaling scale. Based on the first tensor and the second tensor, determine the actual information; Loss information is determined based on the target prediction information and the actual information; The candidate value corresponding to the minimum value among the M loss information is determined as the scaling factor.

9. The method according to any one of claims 1 to 8, characterized in that, The first tensor is the query tensor, and the second tensor is the key tensor; The step of quantizing the scaled first tensor and the scaled second tensor respectively to obtain the quantized first tensor and the quantized second tensor includes: The scaled first tensor and the scaled second tensor are quantized using the per_token flag to obtain the quantized first tensor and the quantized second tensor; or, The scaled first tensor and the scaled second tensor are quantized using tensor per_tensor to obtain the quantized first tensor and the quantized second tensor.

10. The method according to any one of claims 1 to 8, characterized in that, The first tensor is an attention score distribution tensor, and the second tensor is a value tensor, wherein the attention score distribution tensor is determined based on the attention score tensor; The step of quantizing the scaled first tensor and the scaled second tensor respectively to obtain the quantized first tensor and the quantized second tensor includes: The scaled first tensor is quantized per_tensor to obtain the quantized first tensor. The scaled second tensor is quantized according to the per_token tag to obtain the quantized second tensor.

11. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Determine the smoothing scale corresponding to the linear layer in the attention mechanism module; Based on the smoothing scale, the inputs and weights of the linear layer in the attention mechanism module are smoothed; The smoothed input and weights are then quantized.

12. The method according to claim 11, characterized in that, The large model also includes a linear layer, and the method further includes: Determine the smoothing scale corresponding to the j-th linear layer, where j is a positive integer; The input and weights of the j-th linear layer are smoothed according to the smoothing scale. Quantize the input and weights of the j-th linear layer after smoothing; In the attention mechanism module, the smoothing scales of the input projection and the gated projection are equal.

13. The method according to claim 12, characterized in that, If the model structure preceding the linear layer is a normalized layer, the method further includes: The weights of the normalized layer are smoothed according to the smoothing scale of the current linear layer.

14. A large-scale model quantization device, characterized in that, The large model includes an attention mechanism module, and the device includes: The scaling processing module is used to scale the query tensor and the key tensor in the attention mechanism module respectively to obtain scaled query tensor and scaled key tensor, wherein the query tensor and the key tensor are respectively query tensor and key tensor, or the query tensor and the key tensor are respectively key tensor and query tensor. The quantization processing module is used to perform quantization processing on the scaled query tensor and the scaled key tensor respectively to obtain the quantized query tensor and the quantized key tensor. The quantized query tensor and the quantized key tensor are used to determine the output information of the attention mechanism module.

15. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the large model quantization method as described in any one of claims 1 to 13.

Images