Quantitative processing method and performance evaluation method and device of large language model, equipment and medium

Abstract

The invention relates to a quantitative processing method and device for a large language model, a performance evaluation method and device, equipment and a medium. The quantitative processing method comprises the steps that the large language model and quantitative configuration information of all layers to be quantized are acquired; according to the quantization configuration information of each to-be-quantized layer, executing at least one of the following processing on each to-be-quantized layer: quantizing an original weight matrix corresponding to an original linear layer to obtain a quantization weight matrix corresponding to the original linear layer; determining a matched quantization linear layer and replacing the original linear layer with the matched quantization linear layer; determining a matched quantitative attention module and replacing the original attention module with the matched quantitative attention module; and after at least one of the processing is executed on each to-be-quantized layer in the large language model, obtaining a target large language model corresponding to the large language model. Therefore, quantization processing of any to-be-quantized layer of any large language model can be comprehensively, universally and automatically realized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a large language model quantization processing method and performance evaluation method, device and medium. BACKGROUND

[0002] Large language models have achieved remarkable success in the field of natural language processing and have become the current mainstream natural language processing model. Despite their excellent algorithmic performance, large language models with a large number of parameters can result in large storage overhead and high inference latency. Model quantization is a method that can effectively reduce storage overhead and improve inference speed, and has received widespread attention, resulting in a large number of research papers and practical inference engines (such as TensorRT-LLM).

[0003] Considering that a large number of types of series of existing large language models have been developed, such as the LLaMA (Large Language Model Meta Al) series, the OPT (Open Pre-trained Transformer Language Models) series, the GLM (Chatting General Language Model) series, the Bloom (Big Science Large Open-science Open-access Multilingual Language Model) series, the MLA (Multi-head Latent Attention) series, and other large language model series, however, existing technologies usually design corresponding quantization schemes for a single type of large language model, and cannot be applied to the quantization of other large language models, resulting in poor universality and adaptability of model quantization. SUMMARY

[0004] Therefore, the present disclosure proposes a large language model quantization processing method and performance evaluation method, device and medium, which can comprehensively and automatically implement quantization processing for any to-be-quantized layer of any large language model, improve the universality and adaptability of model quantization, and facilitate subsequent implementation of universal model performance evaluation.

[0005] According to an aspect of the present disclosure, a quantization processing method of a large language model is provided, comprising: obtaining a large language model to be quantized and respective quantization configuration information corresponding to each to-be-quantized layer in the large language model, the to-be-quantized layer comprising an original linear layer and / or an original attention module in the large language model, the quantization configuration information comprising a quantization object, bit width information, and data format, the quantization object being used to indicate at least one of an original weight matrix of the original linear layer, an original activation matrix, and an original key-value cache matrix of the original attention module, the bit width information being used to indicate at least one quantization bit width adopted when quantizing the quantization object, and the data format being used to indicate a data format of a quantized object; and performing at least one of the following processing on each to-be-quantized layer in the large language model according to the respective quantization configuration information corresponding to each to-be-quantized layer: in a case where the to-be-quantized layer is the original linear layer and the quantization object comprises the original weight matrix, quantizing the original weight matrix corresponding to the original linear layer according to the bit width information and the data format corresponding to the original linear layer to obtain a quantized weight matrix corresponding to the original linear layer; in a case where the to-be-quantized layer is the original linear layer and the quantization object comprises the original activation matrix, determining a matched quantization linear layer and replacing the original linear layer with the matched quantization linear layer according to the bit width information and the data format corresponding to the original linear layer, the quantization linear layer being a preset linear layer comprising an activation quantization operator, and the activation quantization operator being used to quantize the original activation matrix; in a case where the to-be-quantized layer is the original attention module and the quantization object comprises the original key-value cache matrix, determining a matched quantization attention module and replacing the original attention module with the matched quantization attention module according to the type of the original attention module and the bit width information and the data format corresponding to the original attention module, the quantization attention module being a preset attention module comprising a key-value cache quantization operator and being identical to the type of the original attention module, and the key-value cache quantization operator being used to quantize the original key-value cache matrix; and obtaining a target large language model corresponding to the large language model after performing the at least one processing on each to-be-quantized layer in the large language model, the target large language model being used to implement a natural language processing task.

[0006] In one possible implementation, the quantization configuration information further includes quantization grouping granularity, which indicates whether to perform group quantization on the quantized object and the grouping granularity when performing group quantization on the quantized object in the first channel dimension; wherein, the step of quantizing the original weight matrix corresponding to the original linear layer according to the bit width information and data format corresponding to the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer includes: quantizing the original weight matrix corresponding to the original linear layer according to the bit width information, data format, and quantization grouping granularity corresponding to the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer; the step of quantizing the original weight matrix according to the bit width information corresponding to the original linear layer... The process of determining a matching quantization linear layer and replacing the original linear layer based on the data format includes: determining a matching quantization linear layer based on the quantization bit width, data format, and quantization grouping granularity of the original linear layer; and determining a matching quantization attention module based on the type of the original attention module and its corresponding bit width information and data format, and replacing the original attention module with the matching quantization attention module, includes: determining a matching quantization attention module based on the type of the original attention module and its corresponding bit width information, data format, and quantization grouping granularity, and replacing the original attention module with the matching quantization attention module.

[0007] In one possible implementation, the bit width information corresponding to the layer to be quantized indicates n quantization bit widths, where n is a positive integer and n≥2. The bit width information is in the form of an array to specifically indicate the quantization bit width used by each channel of the quantized object in the second channel dimension.

[0008] In one possible implementation, the number of bits of the weight values ​​in the quantization weight matrix corresponding to the layer to be quantized is less than the number of bits supported by the video memory. After obtaining the quantization weight matrix corresponding to the layer to be quantized, the method further includes: when the bit width information corresponding to the original linear layer indicates one quantization bit width, shifting and concatenating the weight values ​​in the quantization weight matrix corresponding to the original linear layer to obtain weight concatenated data matching the number of bits supported by the video memory, and storing the weight concatenated data in the video memory; when the bit width information corresponding to the original linear layer indicates n quantization bit widths, n≥2, shifting and concatenating the weight values ​​in the quantization weight matrix corresponding to the original linear layer along the second channel dimension to obtain weight concatenated data matching the number of bits supported by the video memory, and storing the weight concatenated data in the video memory.

[0009] According to another aspect of the present disclosure, a performance evaluation method of a large language model is provided, which comprises the quantization processing method, and further comprises: obtaining a target large language model corresponding to the large language model to be evaluated, and performing inference operation of the target large language model to obtain an inference result of the target large language model; wherein the target large language model is obtained by quantizing the large language model to be evaluated by using the quantization processing method; and determining a performance evaluation result of the target large language model based on the inference result of the target large language model.

[0010] In a possible implementation, the target large language model comprises a plurality of Transformer units, each of which comprises at least one of a quantized linear layer and a quantized attention module; and the performance evaluation method further comprises: loading the plurality of Transformer units in the target large language model one by one onto an image processor to perform inference operation until the inference operation of all the Transformer units is completed, so as to obtain the inference result of the target large language model.

[0011] In a possible implementation, the number of bits of the activation value in the quantized activation matrix corresponding to the quantized linear layer in the target large language model is less than the number of bits supported by the display memory, and the quantized activation matrix is obtained by quantizing an original activation matrix input into the quantized linear layer based on corresponding quantization configuration information by an activation quantization operator in the quantized linear layer; and the performance evaluation method further comprises: in a case where the bit width information corresponding to the quantized linear layer indicates one quantization bit width, performing shift splicing on the activation value in the quantized activation matrix corresponding to the quantized linear layer to obtain activation splicing data matching the number of bits supported by the display memory, and storing the activation splicing data in the display memory; and in a case where the bit width information corresponding to the quantized linear layer indicates n quantization bit widths, n≥2, performing shift splicing on the activation value in the quantized activation matrix corresponding to the quantized linear layer along a second channel dimension of the original activation matrix to obtain activation splicing data matching the number of bits supported by the display memory, and storing the activation splicing data in the display memory; wherein the second channel dimension of the original activation matrix is a channel dimension used for indicating the quantization bit widths respectively adopted by each channel in the original activation matrix in the corresponding bit width information.

[0012] In one possible implementation, the number of bits in the key-value cache matrix generated by the quantization attention module in the target large language model is less than the number of bits supported by the video memory. The quantization key-value cache matrix is ​​obtained by quantizing the original key-value cache matrix generated by the quantization attention module based on corresponding quantization configuration information using the key-value cache quantization operator in the quantization attention module. The performance evaluation method further includes: when the bit width information corresponding to the quantization attention module indicates one quantization bit width, shifting and concatenating the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module to obtain a result compatible with the video memory. The system supports key-value concatenation data matching the number of bits supported, and stores the key-value concatenation data in the video memory. When the bit width information corresponding to the quantization attention module indicates n quantization bit widths, and n≥2, the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module are shifted and concatenated along the second channel dimension of the original key-value cache matrix to obtain key-value concatenation data matching the number of bits supported by the video memory, and the key-value concatenation data is stored in the video memory. The second channel dimension of the original key-value cache matrix is ​​the channel dimension in the corresponding bit width information used to indicate the quantization bit width adopted by each channel in the original key-value cache matrix.

[0013] According to another aspect of this disclosure, a performance evaluation apparatus for a large language model is provided, comprising: a quantization processing module for the quantization processing method; and a performance evaluation module for the performance evaluation method.

[0014] According to another aspect of this disclosure, a quantization processing apparatus for a large language model is provided, comprising: an acquisition module, configured to acquire a large language model to be quantized and quantization configuration information corresponding to each layer to be quantized in the large language model, wherein the layer to be quantized includes an original linear layer and / or an original attention module in the large language model, the quantization configuration information includes a quantization object, bit width information, and a data format, wherein the quantization object is used to indicate at least one of the original weight matrix, the original activation matrix, and the original key-value cache matrix of the original linear layer, the bit width information is used to indicate at least one quantization bit width used when quantizing the quantization object, and the data format is used to indicate the data format of the quantized object; and a processing module, configured to perform at least one of the following processing on each layer to be quantized in the large language model according to the quantization configuration information corresponding to each layer to be quantized: when the layer to be quantized is an original linear layer and the quantization object includes an original weight matrix, quantizing the original weight matrix corresponding to the original linear layer according to the bit width information and data format corresponding to the original linear layer, to obtain the original weight matrix corresponding to the original linear layer. The quantization weight matrix is ​​used to determine the matching quantization linear layer and replace the original linear layer with the matching quantization linear layer, based on the bit width information and data format of the original linear layer. The quantization linear layer is a preset linear layer containing an activation quantization operator, which is used to quantize the original activation matrix. When the layer to be quantized is the original attention module and the quantization object includes the original key-value cache matrix, the matching quantization attention module is determined based on the type of the original attention module and its corresponding bit width information and data format, and replaces the original attention module with the matching quantization attention module. The quantization attention module is a preset attention module containing a key-value cache quantization operator and of the same type as the original attention module, which is used to quantize the original key-value cache matrix. An output module is used to obtain a target large language model corresponding to the large language model after performing at least one of the above processes on each layer to be quantized in the large language model. The target large language model is used to implement natural language processing tasks.

[0015] According to another aspect of this disclosure, a performance evaluation apparatus for a large language model is provided, comprising: an inference module, configured to acquire a target large language model corresponding to the large language model to be evaluated, and perform inference operations on the target large language model to obtain an inference result of the target large language model; wherein the target large language model is obtained by quantizing the large language model to be evaluated using the quantization processing method described above; and an evaluation module, configured to determine a performance evaluation result of the target large language model based on the inference result of the target large language model.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described quantization processing method or performance evaluation method when executing instructions stored in the memory.

[0017] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the above-described quantization processing method or performance evaluation method.

[0018] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described quantization processing method or performance evaluation method.

[0019] According to embodiments of this disclosure, by performing at least one of the following processes based on the quantization configuration information of each layer to be quantized in a large language model: quantizing the original weight matrix of the large language model, replacing the original linear layer with a quantized linear layer, and replacing the original attention module with a quantized attention module, comprehensive quantization of weights, activations, and key-value caches can be achieved. That is, quantization processing for any layer to be quantized in any large language model can be achieved automatically and universally, improving the universality and versatility of model quantization, and facilitating subsequent comprehensive and universal model performance evaluation.

[0020] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0021] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0022] Figure 1 A flowchart illustrating a quantization processing method for a large language model according to an embodiment of the present disclosure is shown.

[0023] Figure 2 A schematic diagram of the structure of a Transformer unit according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A schematic diagram of a quantized object according to an embodiment of the present disclosure is shown.

[0025] Figure 4 A flowchart illustrating a performance evaluation method for a large language model according to an embodiment of this disclosure is provided.

[0026] Figure 5 A block diagram of a performance evaluation apparatus for a large language model according to an embodiment of the present disclosure is shown.

[0027] Figure 6 A block diagram of a quantization processing apparatus for a large language model according to an embodiment of the present disclosure is shown.

[0028] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0029] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0030] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0031] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0032] It should be understood that the terms "first," "second," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0033] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0034] In practical applications, the quantization processing method and performance evaluation method of this disclosure can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this application can refer to devices with wireless connection capabilities, which means that they can connect to other terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The terminal devices of this application can also have wired connection communication capabilities. The terminal devices of this application can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking, swiping, etc. on the display screen using fingers, styluses, etc. Non-touchscreen devices can be connected to input devices such as mice, keyboards, and touch panels to control the terminal devices. Screenless devices can be, for example, screenless Bluetooth speakers. For example, the terminal devices of this disclosure can be computers, smartphones, netbooks, tablets, laptops, wearable electronic devices (such as smart bracelets, smartwatches, etc.), TVs, virtual reality devices, etc.

[0035] The quantization processing method and performance evaluation method of this disclosure can also be deployed on a server. This server can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine or container. It has wireless communication capabilities, which can be configured in the server's chip (system) or other components. This can refer to a device with wireless connectivity, meaning it can connect to other servers or terminal devices via Wi-Fi, Bluetooth, or other wireless connection methods. The server of this disclosure can also have wired communication capabilities. For example, the server of this disclosure can be located in the cloud and communicate with terminal devices. The server receives the large language model to be quantized and the quantization configuration information corresponding to each layer to be quantized in the large language model sent by the terminal device. It then uses the quantization processing method deployed on the server to quantize the large language model, obtains the target large language model, and returns it to the terminal device. Finally, it can also use a performance evaluation method not belonging to the server to evaluate the performance of the target large language model, obtain the performance evaluation result of the target large language model, and return it to the terminal device.

[0036] The quantization processing method and performance evaluation method for large language models proposed in the embodiments of this disclosure are described in detail below. It should be understood that the quantization processing method can be performed offline, while the inference performance evaluation method can be performed online.

[0037] Figure 1 A flowchart illustrating a method for quantizing a large language model according to an embodiment of this disclosure is provided. This method can be executed by the aforementioned terminal device or server, such as... Figure 1 As shown, the quantization process includes steps S11 to S13.

[0038] In step S11, the large language model to be quantized and the quantization configuration information corresponding to each layer to be quantized in the large language model are obtained.

[0039] The large language model to be quantized can be any large language model known in the art, such as the LLaMA series, OPT series, GLM series, Bloom series, MLA series, etc. This disclosure does not limit the scope of such large language models.

[0040] The layer to be quantized includes the original linear layer and / or the original attention module in the large language model. The quantization configuration information includes the quantization object, bit width information, and data format. The quantization object is used to indicate at least one of the original weight matrix, the original activation matrix, and the original key-value cache matrix of the original linear layer. The bit width information is used to indicate at least one quantization bit width used when quantizing the quantization object. The data format is used to indicate the data format of the quantized object.

[0041] It should be understood that large language models are typically based on a Transformer architecture. A large language model contains multiple Transformer units, each of which usually consists of a multi-layered structure. For example, Figure 2 The Transformer unit shown mainly includes an attention block, a feedforward network, and a normalization layer, wherein the attention block includes at least a linear layer W. Q W K W V The feedforward neural network includes key-value K-cache and V-cache (also known as KV Cache, where the key-value cache is the intermediate computation result of the attention module) and a multi-headed self-attention layer. It also includes at least a fully connected layer 1 (FC1), an activation layer, and a fully connected layer 2 (FC2). It should be noted that the original linear layers mentioned in this embodiment are the layers used to perform matrix operations in the large language model, such as the normalization layer, the fully connected layer in the feedforward neural network, and the linear layer W in the attention module. Q W K W VBased on this, in the embodiments of this disclosure, if the layer to be quantized is an original linear layer (the original linear layer includes the linear layer in the original attention module), the quantization object can be the original weight matrix and / or the original activation matrix of the original linear layer; if the layer to be quantized is an original attention module, the quantization object can be the original key-value cache matrix in the original attention module. It should be understood that the original key-value cache matrix includes the original K cache matrix and V cache matrix.

[0042] In practical applications, the bit width information corresponding to the layer to be quantized can indicate one type of quantization bit width. In this case, the bit width information can include only a single value indicating the quantization bit width, such as "4". This means that each value in the quantization object (i.e., the entire tensor) of the layer to be quantized (i.e., each weight value in the original weight matrix) is uniformly quantized across the inter-layer dimensions using one type of quantization bit width (e.g., 4 bits). Alternatively, the bit width information can also indicate n types of quantization bit widths, where n is a positive integer and n≥2. This means that a mixed bit width quantization of the intra-layer dimensions is performed on a quantization object, that is, different channels in the quantization object of the same layer to be quantized can be quantized to different bit widths. In order to better indicate the quantization dimension used by each channel in the mixed bit-width quantization of the layer, when the bit-width information corresponding to the layer to be quantized indicates n kinds of quantization bit widths, and n≥2, the bit-width information can be in the form of an array to specifically indicate the quantization bit width used by each channel of the quantized object in the second channel dimension; wherein, the second channel dimension can indicate the "column" or the "row" of the quantized object. Those skilled in the art can set whether the second channel dimension indicates the "column" or the "row" according to actual needs, and this disclosure embodiment does not limit this.

[0043] For example, Figure 3 This diagram illustrates a quantization object, which can be the original weight matrix, the original activation matrix, or the original key-value cache matrix, such as... Figure 3 As shown, if it is desired to quantize the first four channels (channels "T1" to "T4") of a quantized object to 3 bits (i.e., a quantization bit width of 3 bits) and the last four channels (channels "T5" to "T8") to 2 bits (i.e., a quantization bit width of 2 bits) in the column direction, then the bit width information corresponding to the quantized object can be represented as the array "[3,3,3,3,2,2,2,2]"; if it is desired to quantize the first four channels (channels "Q1" to "Q4") of a quantized object to 3 bits and the last four channels (channels "Q5" to "Q8") to 2 bits in the row direction, then the bit width information corresponding to the quantized object can be represented as the array This method allows for the convenient indication of the quantization bit width used by the quantized object in each channel of the second channel dimension using bit width information.

[0044] It should be understood that the bit width information of different layers to be quantized can be different or the same. Those skilled in the art can use the hybrid bit width allocation algorithm known in the art to allocate appropriate quantization bit widths to different layers to be quantized, and can allocate different quantization bit widths to different channels inside the quantization object corresponding to the layer to be quantized. Of course, appropriate quantization bit widths can also be manually defined for different quantization objects and different channels of different importance in the quantization object. The embodiments of this disclosure do not limit the bit width allocation method.

[0045] The data format can include fixed-point or floating-point. The bit width information mentioned above is mainly used to indicate the bit width of quantization (e.g., whether 4-bit or 8-bit is used), and the data format can indicate whether the data format of the quantized object is fixed-point or floating-point. For example, the quantized object (original weight matrix, original activation matrix, and original key-value cache matrix) is usually high-precision floating-point data (e.g., 16-bit floating-point (fp16)). Therefore, the quantized object can be quantized into low-precision fixed-point data (e.g., 4-bit fixed-point data (int4)), in which case the bit width information can indicate a bit width of "4" and the data format can indicate "int". Of course, it can also be quantized into other low-precision floating-point data (e.g., 8-bit floating-point data (fp8)), in which case the bit width information can indicate a bit width of "8" and the data format can indicate "fp". This disclosure does not limit the scope of the embodiments.

[0046] In practical applications, to improve quantization accuracy, embodiments of this disclosure can also support grouped quantization of the quantization object. Therefore, in one possible implementation, the quantization configuration information may further include quantization grouping granularity. This granularity indicates whether to perform grouped quantization of the quantization object and the grouping granularity when performing grouped quantization of the quantization object in the first channel dimension. Grouped quantization involves dividing the quantization object into multiple groups to be quantized according to a preset grouping granularity and quantizing each group separately. The grouping granularity can indicate how many values ​​are contained in each group. For example, the grouping granularity can be represented as m, meaning that the quantization object can be divided into multiple groups of m values, with each group containing m values. The first channel dimension can indicate either the "column" or the "row" of the quantization object. Those skilled in the art can set the first channel dimension to indicate either "column" or "row" according to actual needs; this disclosure does not limit this.

[0047] In this context, multiple groups to be quantized within a quantized object share the same quantization parameter. The quantization parameter may include a scaling factor and / or a quantization zero. The quantization parameter shared by any quantized group is determined based on the m values ​​contained within that group. Specifically, the maximum absolute value among the m values ​​in each quantized group can be determined as the scaling factor for that group, and the minimum absolute value among the m values ​​in each group can be determined as the quantization zero. For example, regarding... Figure 3 The 8×8 quantization object shown can be divided into 16 groups to be quantized in the row direction, with each group consisting of 4 values ​​(e.g., X = {w1, w9, w17, w25} as one group to be quantized), or in the column direction, with each group consisting of 4 values ​​(e.g., X = {w1, w2, w3, w4} as one group to be quantized). Of course, it can also be divided into 8 groups to be quantized in the row or column direction, with each group consisting of 8 values ​​(e.g., X = {w1, w9, w17, w25, w33, w41, w49, w57} or X = {w1, w2, w3, w4, w5, w6, w7, w8} as one group to be quantized). This embodiment of the present disclosure does not limit this. If the absolute value of w9 in the quantized group X = {w1, w9, w17, w25} is the largest and the absolute value of w17 is the smallest, then the scaling factor corresponding to the quantized group X = {w1, w9, w17, w25} can be expressed as absmax(X) = |w9|, and the quantization zero point can be expressed as min(X) = w17. absmax represents taking the maximum absolute value, and min represents taking the minimum absolute value. It should be understood that the quantization parameters of each quantized group can be determined in the above way, so that each quantized group can be quantized based on its corresponding quantization parameters.

[0048] As described above, the bit width information corresponding to the layer to be quantized can indicate one quantization bit width, that is, the entire quantization object adopts one quantization bit width. In this case, if the quantization grouping granularity indicates that the quantization object is grouped for quantization, the grouping granularity m can be set to be divisible by the total number of channels in the first channel dimension, or equal to the total number of channels in the first channel dimension, or any integer value less than the first channel dimension. This disclosure embodiment does not limit this.

[0049] As mentioned above, the bit width information can also indicate n quantization bit widths, where n is a positive integer and n≥2, meaning that the entire quantization object uses n quantization bit widths. In this case, to ensure that the same quantization group divided based on the group granularity uses the same quantization bit width when using group quantization, the second channel dimension and the first channel dimension can be the same. For example, if the bit width information in array form indicates the quantization bit width of each channel in the column direction, then the quantization group granularity during group quantization can indicate the group granularity in the column direction (first channel dimension). Correspondingly, if the bit width information in array form indicates the quantization bit width of each channel in the row direction, then the quantization group granularity during group quantization can indicate the group granularity in the row direction (first channel dimension). In this case, the group granularity m can be set to be divisible by the total number of channels in the first channel dimension, or equal to the total number of channels in the first channel dimension, or any integer value less than the first channel dimension.

[0050] Of course, the second channel dimension can be different from the first channel dimension. However, when the second channel dimension is different from the first channel dimension (e.g., the second channel dimension indicates the column direction while the first channel dimension indicates the row direction), the grouping granularity m should match the bit width information in array form. This ensures that when using grouped quantization, the same quantized group divided based on the grouping granularity uses the same quantization bit width. For example, if the bit width information indicates n quantization bit widths in the second channel dimension, and the quantized object has R channels in the second channel dimension, then the grouping granularity m in the first channel dimension can be set to be divisible by the ratio of R to n (R / n), or set to be equal to R / n, or set to any integer value less than R / n. This ensures that the quantized object divided into the same quantized group has the same quantization bit width. For example, regarding the above... Figure 3 The 8×8 quantization object shown is represented by the array "[3,3,3,3,2,2,2,2]". This indicates that the bit width information indicates two quantization bit widths in the column direction. In this case, the grouping granularity used when grouping the quantization object in the row direction should be less than or equal to 4. For example, it can be divided into 16 groups of quantization (e.g., X = {w1, w9, w17, w25} is one group of quantization) by grouping 4 values ​​each, or it can be divided into 32 groups of quantization (e.g., X = {w1, w9} is one group of quantization) by grouping 2 values ​​each. Then, the quantization parameters corresponding to each group of quantization can be calculated.

[0051] It should be understood that the quantization grouping granularity can also indicate that the quantization object is not grouped for quantization. This situation can be understood as using the same quantization parameter for the entire quantization object. In this case, any known quantization parameter determination algorithm in the art can be used to determine the quantization parameter used for the entire quantization object, or the quantization parameter can be manually set for the quantization object. This disclosure does not limit this.

[0052] In practical applications, since a large language model includes multiple layers to be quantized, data files in a format known in the art (such as .csv or .xls format) can be used to record the quantization configuration information of each layer. This allows for automatic reading of the quantization configuration information of each layer from the data file, thus achieving the acquisition of the quantization configuration information of each layer. It should be understood that those skilled in the art can also use any other known acquisition methods to obtain the quantization configuration information of each layer, and this disclosure does not limit such methods.

[0053] In step S12, based on the quantization configuration information corresponding to each quantization layer, at least one of the following processes is performed on each quantization layer in the large language model:

[0054] Step S121: When the layer to be quantized is the original linear layer and the quantization object includes the original weight matrix, the original weight matrix corresponding to the original linear layer is quantized according to the bit width information and data format of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer.

[0055] Step S122: When the layer to be quantized is the original linear layer and the quantization object includes the original activation matrix, a matching quantization linear layer is determined according to the bit width information and data format corresponding to the original linear layer, and the original linear layer is replaced by the matching quantization linear layer. The quantization linear layer is a preset linear layer containing an activation quantization operator, and the activation quantization operator is used to quantize the original activation matrix.

[0056] Step S123: When the layer to be quantized is the original attention module and the quantization object includes the original key-value cache matrix, a matching quantization attention module is determined according to the type of the original attention module and the corresponding bit width information and data format, and the original attention module is replaced by the matching quantization attention module. The quantization attention module is a preset attention module that contains a key-value cache quantization operator and is of the same type as the original attention module. The key-value cache quantization operator is used to quantize the original key-value cache matrix.

[0057] Step S13: After performing at least one of the above processes on each layer to be quantized in the large language model, the target large language model corresponding to the large language model is obtained. The target large language model is used to implement the natural language processing task.

[0058] In step S121, the original weight matrices of each original linear layer can be quantized offline by traversing all the original linear layers in the large language model, and the quantized weight matrices obtained are then used for model inference. Those skilled in the art can use any known quantization algorithm, such as AWQ (Activation-aware Weight Quantization) or SmoothQuant algorithm, to quantize the original weight matrices. This embodiment does not limit the weight quantization process.

[0059] As described above, the quantization configuration information may also include the quantization grouping granularity. Therefore, in one possible implementation, quantizing the original weight matrix corresponding to the original linear layer according to the bit width information and data format of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer may include: quantizing the original weight matrix corresponding to the original linear layer according to the bit width information, data format, and quantization grouping granularity of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer. Specifically, the original weight matrix corresponding to the original linear layer may first be grouped according to the quantization grouping granularity to obtain multiple groups to be quantized (i.e., weighted groups) divided by the original weight matrix and the quantization parameters corresponding to each group to be quantized are determined. Then, each group to be quantized may be quantized separately according to the bit width information, data format, and quantization parameters corresponding to each group to be quantized to obtain the quantized weight matrix after grouping and quantization. Any quantization algorithm known in the art, such as the AWQ algorithm, SmoothQuant algorithm, etc., may be used to quantize each group to be quantized separately. This disclosure does not limit the implementation of this method.

[0060] Considering that the quantized weight matrix obtained after quantization is usually stored in video memory for subsequent inference, and the number of bits of the weight values ​​in the quantized weight matrix may be less than the number of bits supported by the video memory (for example, the quantized weight value is 8 bits, while the video memory supports 16 bits), in order to save video memory space, after obtaining the quantized weight matrix corresponding to the original linear layer, the method may further include:

[0061] When the bit width information corresponding to the original linear layer indicates a quantization bit width, the weight values ​​in the quantization weight matrix corresponding to the original linear layer are shifted and concatenated to obtain weight concatenated data that matches the number of bits supported by the video memory, and the weight concatenated data is stored in the video memory.

[0062] Given that the bit width information corresponding to the original linear layer indicates n quantization bit widths, and n≥2, the weight values ​​in the quantization weight matrix corresponding to the original linear layer are shifted and concatenated along the second channel dimension to obtain weight concatenated data that matches the number of bits supported by the video memory, and the weight concatenated data is stored in the video memory.

[0063] The bit width information corresponding to the original linear layer indicates that one type of quantization bit width represents uniform bit width quantization within the original weight matrix, meaning that the weight values ​​in the entire quantization weight matrix have the same bit width. Therefore, low-bit weight values ​​can be shifted and concatenated in any channel dimension using shift and concatenation operations, thus solving the problem that the graphics memory hardware does not natively support low-bit data storage and saving graphics memory space. For example, for 2-bit quantization (i.e., the weight values ​​in the quantization weight matrix are 2 bits), shift and concatenation can be performed by shifting and concatenating every 4 2-bit weight values ​​into 1 byte (i.e., concatenating into one weight concatenation data); for 3-bit quantization (i.e., the weight values ​​in the quantization weight matrix are 2 bits), shift and concatenation can be performed by concatenating every 8 3-bit weight values ​​into 3 bytes (i.e., concatenating into 3 weight concatenation data), and so on. This allows for 1 to 16-bit quantization storage based on shift and concatenation operations, achieving graphics memory savings without the need for customized operators for the graphics memory hardware.

[0064] The bit width information corresponding to the original linear layer indicates n quantization bit widths. n≥2 represents intra-layer mixed bit width quantization of the original weight matrix, i.e., channel-by-channel quantization of the original weight matrix. Therefore, to facilitate the shifting and concatenation of weight values ​​with the same bit width, it can be done along the second channel dimension (e.g., ...). Figure 3 The weight values ​​are shifted and concatenated in the column direction (for example, if...) Figure 3 If the first four channels are quantized with 3 bits and the last four channels are quantized with 2 bits, then... Figure 3 The weight values ​​of the first four channels are shifted and concatenated in a way that each set of eight 3-bit weight values ​​is concatenated into three bytes, and the weight values ​​of the last four channels are shifted and concatenated in a way that each set of four 2-bit weight values ​​is concatenated into one byte, thereby achieving memory saving for mixed bit-width quantization within the layer.

[0065] In step S122, by traversing all the original linear layers in the large language model, the original linear layers can be replaced with linear layers containing activation quantization operators. This allows for online quantization of the original activation matrix input to the linear layer during model inference. In practical applications, those skilled in the art can design at least one activation quantization operator based on known quantization algorithms in the field, such as the AWQ algorithm and the SmoothQuant algorithm. Furthermore, based on at least one activation quantization operator, at least one linear layer containing the activation quantization operator can be pre-developed. Different activation quantization operators can correspond to different bit width information and data formats. Therefore, in practical applications, a matching quantized linear layer can be determined based on the bit width information and data format corresponding to the layer to be quantized, and the original buffer layer can be replaced with the matching quantized linear layer. It should be understood that the activation quantization operator in the quantized linear layer matching the original linear layer should match the bit width information and data format corresponding to the layer to be quantized.

[0066] As mentioned above, the quantization configuration information may also include quantization grouping granularity. Different activated quantization operators can correspond to different bit width information, data formats, and quantization grouping granularities. Therefore, in one possible implementation, determining a matching quantization linear layer based on the bit width information and data format corresponding to the layer to be quantized, and replacing the layer to be quantized with the matching quantization linear layer, includes: determining a matching quantization linear layer based on the quantization bit width, data format, and quantization grouping granularity corresponding to the layer to be quantized, and replacing the layer to be quantized with the matching quantization linear layer. It should be understood that the activated quantization operator in the quantization linear layer matching the layer to be quantized should match the bit width information, data format, and quantization grouping granularity corresponding to the layer to be quantized. That is, the activated quantization operator in the quantization linear layer determined based on the quantization bit width, data format, and quantization grouping granularity corresponding to the layer to be quantized has group quantization capability. It should be understood that during the model inference operation, the implementation methods of the activation quantization operator in the quantization linear layer to perform uniform quantization (i.e., using one quantization bit width), mixed quantization (i.e., using two or more quantization bit widths), or group quantization on the input original activation matrix can refer to the specific implementation methods of uniform quantization, mixed quantization, or group quantization on the original weight matrix described above, and will not be elaborated here.

[0067] In step S123, all original attention modules in the large language model can be traversed to replace them with quantized attention modules that contain key-value caching quantization operators and are of the same type as the original attention modules. This allows for better control over the linear layer W in the quantized attention modules during model inference. Q W K W V The resulting raw key-value cache matrix is ​​quantized online.

[0068] As we know, key-value caching is an intermediate computation result of the attention mechanism. Key-value caching quantization differs from weight and activation quantization. Because key-value caching is an intermediate computation result of the attention mechanism, it is usually necessary to modify the source code of the attention module to add a quantization function for the key-value cache. For example, for the transformers library based on the Huggingface framework, the attention class code of mainstream large language models can be extracted and modified, and a quantization function can be introduced into the forward function to quantize the key-value cache online. Although this method can support all models in the transformers library, it requires modification of the model inference source code to support it. Therefore, to reduce the modification cost, multiple quantization attention modules with key-value caching quantization operators can be pre-designed (i.e., pre-design some attention module templates) so that they can be automatically called when key-value caching quantization is needed.

[0069] It should be understood that the structure of attention modules in different types of large language models typically differs. For example, based on the type of attention operation, existing large language models can be categorized into five major series: LLaMA, OPT, GLM, Bloom, and MLA (i.e., five types of attention modules). Furthermore, various quantization attention modules incorporating key-value cached quantization operators can be designed for these five series. It should also be understood that those skilled in the art can design at least one key-value cached quantization operator based on known quantization algorithms such as AWQ and SmoothQuant. Based on this key-value cached quantization operator and different types of attention modules in various large language models, multiple quantization attention modules can be designed. This allows for the automatic selection and replacement of the original attention module with a matching quantization attention module based on its type within the large language model to be quantized. Different key-value cached quantization operators can correspond to different bit widths, data formats, and quantization grouping granularities. Therefore, a matching quantization attention module can be determined based on the bit width information and data format of the layer to be quantized, and the original attention module can be replaced with the matching quantization attention module. It should be understood that the key-value cache quantization operator in the quantization attention module that matches the original attention module should match the bit width information and data format of the original attention module.

[0070] As described above, the quantization configuration information may also include the quantization grouping granularity. Therefore, in one possible implementation, determining a matching quantization attention module based on the type of the original attention module containing the layer to be quantized, as well as the bit width information and data format corresponding to the layer to be quantized, and replacing the original attention module containing the layer to be quantized with the matching quantization attention module includes: determining a matching quantization attention module based on the type of the original attention module containing the layer to be quantized, as well as the bit width information, data format, and quantization grouping granularity corresponding to the layer to be quantized, and replacing the original attention module containing the layer to be quantized with the matching quantization attention module. It should be understood that the key-value cache quantization operator in the quantization attention module matching the original attention module should match the bit width information, data format, and quantization grouping granularity corresponding to the original attention module; that is, the key-value cache quantization operator in the quantization attention module determined based on the quantization bit width, data format, and quantization grouping granularity corresponding to the original attention module has group quantization capability. It should be understood that during the model inference operation, the implementation of the key-value cache quantization operator in the quantization attention module for uniform quantization (i.e., using one quantization bit width), mixed quantization (i.e., using two or more quantization bit widths), or group quantization of the original key-value cache matrix generated by the linear layer in the quantization attention module can refer to the specific implementation methods for uniform quantization, mixed quantization, or group quantization of the original weight matrix described above, and will not be elaborated here.

[0071] Considering that more types of attention modules may be developed as technology advances, the method of this disclosure can be applied to existing open-source libraries, such as the Huggingface framework. By aligning with the relevant parameters in the Huggingface framework, it is possible to easily support new types of attention modules, thereby enabling the extension of key-value cache quantization for different attention modules.

[0072] According to the quantization processing method of this disclosure, by performing at least one of the following processes based on the quantization configuration information of each layer to be quantized in a large language model: quantizing the original weight matrix of the large language model, replacing the original linear layer with a quantized linear layer, and replacing the original attention module with a quantized attention module, comprehensive quantization of weights, activations, and key-value caches can be achieved. That is, it can automatically and universally realize quantization processing for any layer to be quantized in any large language model, improve the universality and versatility of model quantization, and facilitate subsequent comprehensive and universal model performance evaluation.

[0073] Given the current lack of comprehensive performance evaluation techniques to assess the impact of model quantization on the performance of large language model algorithms, existing techniques such as ZeroQuant V2, while evaluating quantization weights and activations, only perform evaluations on a limited number of natural language understanding tasks. QEmergent only evaluates the impact of quantization weights and activations on the Emergent capability of large language models. In other words, existing techniques like ZeroQuant V2 and QEmergent only quantize weights and activations, evaluate a limited number of tasks, and evaluate a limited number of models. Furthermore, neither has a comprehensive open-source quantization and evaluation framework, hindering the evaluation of more models, tasks, and tensor types in the future.

[0074] Therefore, based on the above-described quantification method, embodiments of this disclosure also provide Figure 4 The performance evaluation method for large language models shown includes the quantization processing method described in the embodiments of this disclosure above, such as... Figure 4 As shown, the performance evaluation method further includes:

[0075] Step S24: Obtain the target large language model corresponding to the large language model to be evaluated, and perform inference operations on the target large language model to obtain the inference result of the target large language model; wherein, the target large language model is obtained by quantizing the large language model to be evaluated using the quantization processing method described in the above embodiments of this disclosure.

[0076] Step S25: Based on the inference results of the target large language model, determine the performance evaluation results of the target large language model.

[0077] In practical applications, those skilled in the art can design question-answer pair data for evaluating the performance of the target large language model, based on the application scenario of the model. This question-answer pair data includes questions and reference answers, such as common-sense questions and reference answers in natural language descriptions, navigation questions and reference navigation results, path planning questions and reference planning results, etc. Thus, during the inference operation of the target large language model, questions from the question-answer pair data can be input into the model to obtain the inference result output by the model (i.e., the predicted answer output by the model). Then, the performance of the target large language model can be evaluated based on the difference between the reference answer to the input question and the predicted answer output by the model.

[0078] Those skilled in the art can use any known performance evaluation metrics in the field, such as accuracy, recall, success rate, etc., to achieve inference results based on the target large language model and determine the performance evaluation results of the target large language model. This disclosure does not limit such inference results.

[0079] Understandably, the target large language model comprises multiple Transformer units, each of which may include at least one of a quantized linear layer and a quantized attention module. Considering the limited hardware resources available for performing model inference and the significant resource overhead required for model inference (such as substantial memory consumption), to save resources, in one possible implementation, the aforementioned inference operation of the target large language model to obtain the inference result can include: loading the multiple Transformer units in the target large language model one by one onto the GPU for inference operations until the inference operations of all Transformer units are completed, thus obtaining the inference result of the target large language model. This method can be understood as loading only one Transformer unit onto the GPU for inference at a time, and loading the next Transformer unit onto the GPU only after the inference of the current Transformer unit has ended. Compared to loading the entire model onto the GPU at once, this avoids the significant memory overhead.

[0080] It should be understood that during the inference operation of the target large language model, the number of bits of the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer in the target large language model may be less than the number of bits supported by the video memory. The quantization activation matrix is ​​obtained by quantizing the original activation matrix input to the quantization linear layer using the activation quantization operator in the quantization linear layer based on the corresponding quantization configuration information (i.e., bit width information, data format, and quantization grouping granularity). The performance evaluation method further includes:

[0081] When the bit width information corresponding to the quantization linear layer indicates a quantization bit width, the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer are shifted and concatenated to obtain activation concatenated data that matches the number of bits supported by the video memory, and the activation concatenated data is stored in the video memory.

[0082] When the bit width information corresponding to the quantization linear layer indicates n quantization bit widths, and n≥2, the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer are shifted and concatenated along the second channel dimension of the original activation matrix to obtain activation concatenation data that matches the number of bits supported by the video memory, and the activation concatenation data is stored in the video memory; wherein, the second channel dimension of the original activation matrix is ​​the channel dimension in the bit width information corresponding to the quantization linear layer used to indicate the quantization bit width adopted by each channel in the original activation matrix.

[0083] Specifically, the method described above for shifting and concatenating the quantization weight matrix can be used to shift and concatenate the activation values ​​in the quantization activation matrix (this method can be understood as shifting and concatenating activation values ​​in any channel dimension, or concatenating activation values ​​according to rows or columns); and, referring to the method described above for shifting and concatenating the quantization weight matrix, the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer can be shifted and concatenated along the second channel dimension of the original activation matrix (this method can be understood as shifting and concatenating activation values ​​according to the channel dimension allocated by the bit width), which will not be elaborated here.

[0084] It should be understood that during the inference operation of the target large language model, the number of bits of the key-value cache value in the quantization key-value cache matrix generated by the quantization attention module in the target large language model may be less than the number of bits supported by the video memory. The quantization key-value cache matrix is ​​obtained by quantizing the original key-value cache matrix generated in the quantization attention module based on the corresponding quantization configuration information by the key-value cache quantization operator in the quantization attention module (that is, by quantizing the original key-value cache matrix generated by the linear layer in the quantization attention module); wherein, the performance evaluation method further includes:

[0085] When the bit width information corresponding to the quantization attention module indicates a quantization bit width, the key value cache value in the quantization key value cache matrix generated by the quantization attention module is shifted and concatenated to obtain key value concatenated data that matches the number of bits supported by the video memory, and the key value concatenated data is stored in the video memory.

[0086] When the bit width information corresponding to the quantization attention module indicates n quantization bit widths, and n≥2, the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module are shifted and concatenated along the second channel dimension of the original key-value cache matrix to obtain key-value concatenated data that matches the number of bits supported by the video memory, and the key-value concatenated data is stored in the video memory; wherein, the second channel dimension of the original key-value cache matrix is ​​the channel dimension in the corresponding bit width information used to indicate the quantization bit width adopted by each channel in the original key-value cache matrix.

[0087] Specifically, the method described above for shifting and concatenating the quantization weight matrix can be used to shift and concatenate the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module (this method can be understood as shifting and concatenating key-value cache values ​​in any channel dimension, or in other words, shifting and concatenating key-value cache values ​​according to rows or columns); and, the method described above for shifting and concatenating the quantization weight matrix can be used to shift and concatenate the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module along the second channel dimension of the original key-value cache matrix (this method can be understood as shifting and concatenating key-value cache values ​​according to the channel dimension that implements bit width allocation within the original key-value cache matrix), which will not be elaborated here.

[0088] The performance evaluation method according to the embodiments of this disclosure can universally and comprehensively achieve performance evaluation of the target large language model obtained after quantization of any large language model.

[0089] Based on the quantization processing method and performance evaluation method provided in the embodiments of this disclosure above, the embodiments of this disclosure also provide a method such as... Figure 5 The performance evaluation apparatus shown includes: a quantization processing module 501, used to execute the quantization processing method of the present disclosure embodiment described above; and a performance evaluation module 502, used to execute the performance evaluation method of the present disclosure embodiment described above.

[0090] The performance evaluation apparatus of this disclosure supports the quantization of weights, activations, and key-value caches in model quantization, and can also integrate mainstream reparameterization-based quantization algorithms (such as AWQ and SmoothQuant). The quantization processing module 501 can be packaged into a simple interface within the evaluation apparatus for easy invocation. This interface can include input parameters for four aspects: quantization bit width, quantization object, quantization grouping granularity, and data format. This allows the quantization processing module 501 to be integrated into any existing evaluation framework with a single line of code, enabling seamless input of the quantized model to the performance evaluation module 502 for evaluation.

[0091] Furthermore, in terms of model performance evaluation, existing evaluation frameworks such as OpenCompass and lm-evaluation-harness generally do not natively support model quantization and do not support performance evaluation of quantized large language models on resource-constrained hardware. This mainly presents two problems: (1) they do not natively support the computational data flow of quantized models; (2) the memory overhead of large language model evaluation is enormous, making it difficult to fully accommodate on resource-constrained hardware. In view of this, the performance evaluation device provided in this embodiment can separate model quantization from model evaluation, achieving simple integration of model quantization without affecting the evaluation code. It can easily utilize most existing open-source evaluation frameworks for performance evaluation of quantized models. Specifically, for cases where the computational data flow of quantized models is not natively supported, the performance evaluation device in this embodiment can provide a set of interface definitions for the quantization processing module 501, which can be plugged and played in existing evaluation frameworks. When passing interface parameters, only the above four input parameters need to be uniformly passed to complete model quantization and evaluation, and it can be widely applied in various evaluation frameworks. To address the issue of huge memory overhead in evaluating large language models, one can load only one Transformer unit onto the GPU for inference at a time, and load the next Transformer unit onto the GPU only after the current module's Transformer unit has finished, thus avoiding huge memory overhead.

[0092] Based on the quantization processing method of the above embodiments of this disclosure, Figure 6 A block diagram of a quantization processing apparatus for a large language model according to an embodiment of the present disclosure is shown, such as Figure 6 As shown, the device includes:

[0093] The acquisition module 601 is used to acquire the large language model to be quantized and the quantization configuration information corresponding to each layer to be quantized in the large language model. The layer to be quantized includes the original linear layer and / or the original attention module in the large language model. The quantization configuration information includes the quantization object, bit width information and data format. The quantization object is used to indicate at least one of the original weight matrix, the original activation matrix and the original key-value cache matrix of the original linear layer and the original attention module. The bit width information is used to indicate at least one quantization bit width used when quantizing the quantization object. The data format is used to indicate the data format of the quantized object.

[0094] Processing module 602 is configured to perform at least one of the following processes on each of the layers to be quantized in the large language model according to the quantization configuration information corresponding to each layer:

[0095] When the layer to be quantized is the original linear layer and the quantization object includes the original weight matrix, the original weight matrix corresponding to the original linear layer is quantized according to the bit width information and data format of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer.

[0096] When the layer to be quantized is the original linear layer and the quantization object includes the original activation matrix, a matching quantization linear layer is determined according to the bit width information and data format corresponding to the original linear layer, and the original linear layer is replaced by the matching quantization linear layer. The quantization linear layer is a preset linear layer containing an activation quantization operator, and the activation quantization operator is used to quantize the original activation matrix.

[0097] When the layer to be quantized is the original attention module and the quantization object includes the original key-value cache matrix, a matching quantization attention module is determined according to the type of the original attention module and the corresponding bit width information and data format, and the original attention module is replaced by the matching quantization attention module. The quantization attention module is a preset attention module that contains a key-value cache quantization operator and is of the same type as the original attention module. The key-value cache quantization operator is used to quantize the original key-value cache matrix.

[0098] The output module 603 is used to obtain a target large language model corresponding to the large language model after performing at least one of the above processes on each layer to be quantized in the large language model. The target large language model is used to implement natural language processing tasks.

[0099] In one possible implementation, the quantization configuration information further includes quantization grouping granularity, which indicates whether to perform group quantization on the quantized object and the grouping granularity when performing group quantization on the quantized object in the first channel dimension; wherein, the step of quantizing the original weight matrix corresponding to the original linear layer according to the bit width information and data format corresponding to the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer includes: quantizing the original weight matrix corresponding to the original linear layer according to the bit width information, data format, and quantization grouping granularity corresponding to the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer; the step of quantizing the original weight matrix according to the bit width information corresponding to the original linear layer... The process of determining a matching quantization linear layer and replacing the original linear layer based on the data format includes: determining a matching quantization linear layer based on the quantization bit width, data format, and quantization grouping granularity of the original linear layer; and determining a matching quantization attention module based on the type of the original attention module and its corresponding bit width information and data format, and replacing the original attention module with the matching quantization attention module, includes: determining a matching quantization attention module based on the type of the original attention module and its corresponding bit width information, data format, and quantization grouping granularity, and replacing the original attention module with the matching quantization attention module.

[0100] In one possible implementation, the bit width information corresponding to the layer to be quantized indicates n quantization bit widths, where n is a positive integer and n≥2. The bit width information is in the form of an array to specifically indicate the quantization bit width used by each channel of the quantized object in the second channel dimension.

[0101] In one possible implementation, the number of bits of the weight values ​​in the quantization weight matrix is ​​less than the number of bits supported by the video memory. After obtaining the quantization weight matrix corresponding to the original linear layer, the method further includes: when the bit width information corresponding to the original linear layer indicates one quantization bit width, shifting and concatenating the weight values ​​in the quantization weight matrix corresponding to the original linear layer to obtain weight concatenated data matching the number of bits supported by the video memory, and storing the weight concatenated data in the video memory; when the bit width information corresponding to the original linear layer indicates n quantization bit widths, n≥2, shifting and concatenating the weight values ​​in the quantization weight matrix corresponding to the original linear layer along the second channel dimension to obtain weight concatenated data matching the number of bits supported by the video memory, and storing the weight concatenated data in the video memory.

[0102] According to the quantization evaluation apparatus of this disclosure, by performing at least one of the following processes based on the quantization configuration information of each layer to be quantized in a large language model: quantizing the original weight matrix of the large language model, replacing the original linear layer with a quantized linear layer, and replacing the original attention module with a quantized attention module, it can achieve comprehensive quantization of weights, activations, and key-value caches. That is, it can automatically and universally achieve quantization processing for any layer to be quantized in any large language model, improve the universality and versatility of model quantization, and facilitate subsequent comprehensive and universal model performance evaluation.

[0103] This disclosure also provides a performance evaluation device for a large language model, comprising: an inference module, configured to acquire a target large language model corresponding to the large language model to be evaluated, and perform inference operations on the target large language model to obtain an inference result of the target large language model; wherein the target large language model is obtained by quantizing the large language model to be evaluated using the quantization processing method described above; and an evaluation module, configured to determine a performance evaluation result of the target large language model based on the inference result of the target large language model.

[0104] In one possible implementation, the target large language model includes multiple Transformer units, each Transformer unit including at least one of a quantized linear layer and a quantized attention module; wherein, performing inference operations on the target large language model to obtain the inference result of the target large language model includes: loading the multiple Transformer units in the target large language model one by one onto an image processor for inference operations until the inference operations of all Transformer units are completed, thereby obtaining the inference result of the target large language model.

[0105] In one possible implementation, the number of bits of activation values ​​in the quantization activation matrix corresponding to the quantization linear layer in the target large language model is less than the number of bits supported by the video memory. The quantization activation matrix is ​​obtained by quantizing the original activation matrix input to the quantization linear layer using the activation quantization operator in the quantization linear layer based on the corresponding quantization configuration information. The performance evaluation device further includes an activation shift and concatenation module, used to: when the bit width information corresponding to the quantization linear layer indicates a quantization bit width, shift and concatenate the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer to obtain a value that matches the number of bits supported by the video memory. The activation concatenation data that matches the number of bits supported is stored in the video memory. When the bit width information corresponding to the quantization linear layer indicates n quantization bit widths, and n≥2, the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer are shifted and concatenated along the second channel dimension of the original activation matrix to obtain activation concatenation data that matches the number of bits supported by the video memory, and the activation concatenation data is stored in the video memory. The second channel dimension of the original activation matrix is ​​the channel dimension in the corresponding bit width information used to indicate the quantization bit width adopted by each channel in the original activation matrix.

[0106] In one possible implementation, the number of bits in the key-value cache matrix generated by the quantization attention module in the target large language model is less than the number of bits supported by the video memory. The quantization key-value cache matrix is ​​obtained by quantizing the original key-value cache matrix generated by the quantization attention module based on corresponding quantization configuration information using the key-value cache quantization operator in the quantization attention module. The performance evaluation device further includes a key-value cache shifting and concatenating module, used to shift and concatenate the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module when the bit width information corresponding to the quantization attention module indicates a quantization bit width, to obtain... The process involves obtaining key-value concatenation data that matches the number of bits supported by the video memory, and storing the key-value concatenation data in the video memory. If the bit width information corresponding to the quantization attention module indicates n quantization bit widths, and n≥2, the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module are shifted and concatenated along the second channel dimension of the original key-value cache matrix to obtain key-value concatenation data that matches the number of bits supported by the video memory, and the key-value concatenation data is stored in the video memory. The second channel dimension of the original key-value cache matrix is ​​the channel dimension in the corresponding bit width information used to indicate the quantization bit width adopted by each channel in the original key-value cache matrix.

[0107] The performance evaluation apparatus according to the embodiments of this disclosure can universally and comprehensively realize the performance evaluation of the target large language model obtained after quantization of any large language model.

[0108] In some embodiments, the framework and apparatus provided in this disclosure may have functions or include modules that can be used to perform the methods described in the above method embodiments. The specific implementation of these methods can be referred to the description of the above method embodiments, and for the sake of brevity, they will not be repeated here.

[0109] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0110] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0111] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0112] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, device 1900 may be provided as a server or terminal device. (Refer to...) Figure 7 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0113] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0114] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0115] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0116] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0117] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0118] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0119] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0121] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0123] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A quantization processing method for a large language model, characterized in that, include: Obtain the large language model to be quantized and the quantization configuration information corresponding to each layer to be quantized in the large language model. The layers to be quantized include the original linear layer and / or the original attention module in the large language model. The quantization configuration information includes the quantization object, bit width information, and data format. The quantization object is used to indicate at least one of the original weight matrix, the original activation matrix, and the original key-value cache matrix of the original linear layer. The bit width information is used to indicate at least one quantization bit width used when quantizing the quantization object. The data format is used to indicate the data format of the quantized object. Based on the quantization configuration information corresponding to each layer to be quantized, at least one of the following processes are performed on each layer to be quantized in the large language model: When the layer to be quantized is the original linear layer and the quantization object includes the original weight matrix, the original weight matrix corresponding to the original linear layer is quantized according to the bit width information and data format of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer. When the layer to be quantized is the original linear layer and the quantization object includes the original activation matrix, a matching quantization linear layer is determined according to the bit width information and data format corresponding to the original linear layer, and the original linear layer is replaced by the matching quantization linear layer. The quantization linear layer is a preset linear layer containing an activation quantization operator, and the activation quantization operator is used to quantize the original activation matrix. When the layer to be quantized is the original attention module and the quantization object includes the original key-value cache matrix, a matching quantization attention module is determined according to the type of the original attention module and the corresponding bit width information and data format, and the original attention module is replaced by the matching quantization attention module. The quantization attention module is a preset attention module that contains a key-value cache quantization operator and is of the same type as the original attention module. The key-value cache quantization operator is used to quantize the original key-value cache matrix. After performing at least one of the above processes on each layer to be quantized in the large language model, a target large language model corresponding to the large language model is obtained, and the target large language model is used to implement natural language processing tasks.

2. The method according to claim 1, characterized in that, The quantization configuration information also includes quantization grouping granularity, which is used to indicate whether to group the quantization object and the grouping granularity when grouping the quantization object in the first channel dimension. The step of quantizing the original weight matrix corresponding to the original linear layer according to the bit width information and data format of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer includes: quantizing the original weight matrix corresponding to the original linear layer according to the bit width information, data format and quantization grouping granularity of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer. The step of determining a matching quantization linear layer and replacing the original linear layer with the matching quantization linear layer based on the bit width information and data format of the original linear layer includes: determining a matching quantization linear layer and replacing the original linear layer with the matching quantization linear layer based on the quantization bit width, data format and quantization grouping granularity of the original linear layer. The step of determining a matching quantization attention module based on the type of the original attention module and its corresponding bit width information and data format, and replacing the original attention module with the matching quantization attention module, includes: determining a matching quantization attention module based on the type of the original attention module and its corresponding bit width information, data format and quantization grouping granularity, and replacing the original attention module with the matching quantization attention module.

3. The method according to claim 1 or 2, characterized in that, The bit width information corresponding to the layer to be quantized indicates n kinds of quantization bit widths, where n is a positive integer. When n≥2, the bit width information is in the form of an array to specifically indicate the quantization bit width used by each channel of the quantization object in the second channel dimension.

4. The method according to claim 3, characterized in that, The number of bits for the weight values ​​in the quantization weight matrix is ​​less than the number of bits supported by the video memory. After obtaining the quantization weight matrix corresponding to the original linear layer, the method further includes: When the bit width information corresponding to the original linear layer indicates a quantization bit width, the weight values ​​in the quantization weight matrix corresponding to the original linear layer are shifted and concatenated to obtain weight concatenated data that matches the number of bits supported by the video memory, and the weight concatenated data is stored in the video memory. Given that the bit width information corresponding to the original linear layer indicates n quantization bit widths, and n≥2, the weight values ​​in the quantization weight matrix corresponding to the original linear layer are shifted and concatenated along the second channel dimension to obtain weight concatenated data that matches the number of bits supported by the video memory, and the weight concatenated data is stored in the video memory.

5. A performance evaluation method for a large language model, characterized in that, include: Obtain the target large language model corresponding to the large language model to be evaluated, and perform inference operations on the target large language model to obtain the inference result of the target large language model; wherein, the target large language model is obtained by quantizing the large language model to be evaluated using the quantization processing method according to any one of claims 1 to 4; Based on the inference results of the target large language model, the performance evaluation results of the target large language model are determined.

6. The performance evaluation method according to claim 5, characterized in that, The target large language model includes multiple Transformer units, and each Transformer unit includes at least one of a quantized linear layer and a quantized attention module. The step of performing inference operations on the target large language model to obtain the inference result of the target large language model includes: Multiple Transformer units in the target large language model are loaded one by one onto the image processor for inference operations until the inference operations of all Transformer units are completed, thereby obtaining the inference result of the target large language model.

7. The performance evaluation method according to claim 5 or 6, characterized in that, In the target large language model, the number of bits of activation values ​​in the quantization activation matrix corresponding to the quantization linear layer is less than the number of bits supported by the video memory. The quantization activation matrix is ​​obtained by quantizing the original activation matrix input to the quantization linear layer using the activation quantization operator in the quantization linear layer based on the corresponding quantization configuration information. The performance evaluation method further includes: When the bit width information corresponding to the quantization linear layer indicates a quantization bit width, the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer are shifted and concatenated to obtain activation concatenated data that matches the number of bits supported by the video memory, and the activation concatenated data is stored in the video memory. When the bit width information corresponding to the quantization linear layer indicates n quantization bit widths, and n≥2, the activation values ​​in the quantization activation matrix corresponding to the quantization linear layer are shifted and concatenated along the second channel dimension of the original activation matrix to obtain activation concatenated data that matches the number of bits supported by the video memory, and the activation concatenated data is stored in the video memory; wherein, the second channel dimension of the original activation matrix is ​​the channel dimension in the corresponding bit width information used to indicate the quantization bit width adopted by each channel in the original activation matrix.

8. The performance evaluation method according to claim 5 or 6, characterized in that, The number of bits in the key-value cache matrix generated by the quantization attention module in the target large language model is less than the number of bits supported by the video memory. The quantization key-value cache matrix is ​​obtained by quantizing the original key-value cache matrix generated by the quantization attention module based on the corresponding quantization configuration information using the key-value cache quantization operator in the quantization attention module. The performance evaluation method further includes: When the bit width information corresponding to the quantization attention module indicates a quantization bit width, the key value cache value in the quantization key value cache matrix generated by the quantization attention module is shifted and concatenated to obtain key value concatenated data that matches the number of bits supported by the video memory, and the key value concatenated data is stored in the video memory. When the bit width information corresponding to the quantization attention module indicates n quantization bit widths, and n≥2, the key-value cache values ​​in the quantization key-value cache matrix generated by the quantization attention module are shifted and concatenated along the second channel dimension of the original key-value cache matrix to obtain key-value concatenated data that matches the number of bits supported by the video memory, and the key-value concatenated data is stored in the video memory; wherein, the second channel dimension of the original key-value cache matrix is ​​the channel dimension in the corresponding bit width information used to indicate the quantization bit width adopted by each channel in the original key-value cache matrix.

9. A performance evaluation device for a large language model, characterized in that, include: A quantization processing module is used to execute the quantization processing method according to any one of claims 1 to 4; A performance evaluation module for performing the performance evaluation method according to any one of claims 5 to 8.

10. A quantization processing device for a large language model, characterized in that, include: The acquisition module is used to acquire the large language model to be quantized and the quantization configuration information corresponding to each layer to be quantized in the large language model. The layer to be quantized includes the original linear layer and / or the original attention module in the large language model. The quantization configuration information includes the quantization object, bit width information and data format. The quantization object is used to indicate at least one of the original weight matrix, the original activation matrix of the original linear layer and the original key-value cache matrix of the original attention module. The bit width information is used to indicate at least one quantization bit width used when quantizing the quantization object. The data format is used to indicate the data format of the quantized object. The processing module is configured to perform at least one of the following processes on each of the layers to be quantized in the large language model, based on the quantization configuration information corresponding to each layer: When the layer to be quantized is the original linear layer and the quantization object includes the original weight matrix, the original weight matrix corresponding to the original linear layer is quantized according to the bit width information and data format of the original linear layer to obtain the quantized weight matrix corresponding to the original linear layer. When the layer to be quantized is the original linear layer and the quantization object includes the original activation matrix, a matching quantization linear layer is determined according to the bit width information and data format corresponding to the original linear layer, and the original linear layer is replaced by the matching quantization linear layer. The quantization linear layer is a preset linear layer containing an activation quantization operator, and the activation quantization operator is used to quantize the original activation matrix. When the layer to be quantized is the original attention module and the quantization object includes the original key-value cache matrix, a matching quantization attention module is determined according to the type of the original attention module and the corresponding bit width information and data format, and the original attention module is replaced by the matching quantization attention module. The quantization attention module is a preset attention module that contains a key-value cache quantization operator and is of the same type as the original attention module. The key-value cache quantization operator is used to quantize the original key-value cache matrix. The output module is used to obtain a target large language model corresponding to the large language model after performing at least one of the above processes on each layer to be quantized in the large language model. The target large language model is used to implement natural language processing tasks.

11. A performance evaluation device for a large language model, characterized in that, include: The reasoning module is used to obtain the target large language model corresponding to the large language model to be evaluated, and to perform reasoning operations on the target large language model to obtain the reasoning result of the target large language model; wherein, the target large language model is obtained by quantizing the large language model to be evaluated using the quantization processing method according to any one of claims 1 to 4. The evaluation module is used to determine the performance evaluation result of the target large language model based on the reasoning results of the target large language model.

12. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the quantization processing method according to any one of claims 1 to 4 or the performance evaluation method according to any one of claims 5 to 8 when executing instructions stored in the memory.

13. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the quantization processing method according to any one of claims 1 to 4 or the performance evaluation method according to any one of claims 5 to 8.

Images