Model quantization method, electronic device, and storage medium

Abstract

The application provides a model quantization method, an electronic device and a storage medium, and relates to the technical field of computers. By using at least two quantization bit widths to quantize linear layer weights of a model, the problem that all linear layer weights of the model are quantized using low quantization bit widths, resulting in serious loss of model accuracy, or the problem that the model occupies a large amount of memory due to quantization using high quantization bit widths, is solved. The method comprises the following steps: quantizing linear layer weights of a model based on multiple sets of quantization data to obtain multiple candidate models; testing the performance of the multiple candidate models using the same test set to determine the performance indicators of each candidate model; and after the quantization data corresponding to a target model in the multiple candidate models, which satisfies a preset condition, is taken as target quantization data, quantizing the model using the target quantization data to obtain a quantized model. One set of quantization data is used to obtain one candidate model, and the quantization bit widths of at least two linear layer weights are different.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model quantization method, electronic device, and storage medium. Background Technology

[0002] A model refers to a deep learning model with a very large number of parameters. These models are typically used to process large-scale datasets and learn complex features of the data. Due to the large number of parameters, when deploying models on the terminal side, it is usually necessary to perform model quantization and compression to reduce the deployment difficulty.

[0003] Quantization compression of a model refers to the process of converting the model's weights from floating-point numbers to integers or floating-point numbers with a lower bit width. How to effectively quantize and compress a model while avoiding severe loss of accuracy is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a model quantization method, electronic device, and storage medium. By quantizing the linear layer weights of the model using at least two quantization bit widths, the problem of severe loss of model accuracy caused by quantizing all linear layer weights of the model using a low quantization bit width is reduced.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a model quantization method is provided for application in electronic devices, the method comprising:

[0007] The electronic device quantizes the linear layer weights of a model based on multiple sets of quantized data to obtain multiple candidate models. Then, it performs performance tests on these candidate models using the same test set to determine the performance metrics of each candidate model. The quantized data corresponding to the target model whose performance metrics meet preset conditions is then used as the target quantized data for that model. Next, the electronic device quantizes the model based on the target quantized data to obtain the quantized model. Specifically, one set of quantized data is used to obtain one candidate model, and another set of quantized data is used to indicate the quantization bit width of the linear layer weights of the model; at least two linear layer weights have different quantization bit widths.

[0008] Based on the method described in the first aspect, the electronic device solves the problems of severe accuracy loss caused by using the same low quantization bit width for all linear layer weights, or insufficient model compression and large model size caused by using the same high quantization bit width for all linear layer weights. Furthermore, by using the same test set to perform performance tests on multiple candidate models, the electronic device can more realistically test the performance indicators of multiple candidate models, thereby more accurately determining the target model whose performance indicators meet the preset conditions. This results in target quantized data with minimal accuracy loss and reduced model size during quantization compression.

[0009] In one possible implementation of the first aspect, the above method may further include:

[0010] The electronic device performs multiple adjustments to the initial quantization bit width of the linear layer weights of the model, resulting in multiple sets of quantized data.

[0011] In this process, an adjustment operation yields a set of quantized data. The adjustment operation includes adjusting the quantization bit width of the first part of the linear layer weights in the model to the first quantization bit width, adjusting the quantization bit width of the second part of the linear layer weights to the second quantization bit width, and adjusting the quantization bit width of the third part of the linear layer weights to the third quantization bit width, until the difference between the average bit width of all adjusted linear layer weights and the preset average bit width is within the preset range; the first quantization bit width is greater than the second quantization bit width, and the second quantization bit width is greater than the third quantization bit width.

[0012] Based on this implementation, the electronic device can obtain multiple sets of quantized data by adjusting the initial quantization bit width of the linear layer weights multiple times. Each set of quantized data satisfies that the difference between the average bit width and the preset average bit width is within a preset range. This allows the electronic device to subsequently select the target quantized data from the multiple sets of quantized data, which results in minimal accuracy loss and volume compression when quantizing and compressing the model.

[0013] In another possible implementation of the first aspect, the above method may further include:

[0014] The electronic device sorts the linear layer weights in the model according to a preset sorting rule, and sets the initial quantization bit width corresponding to the sorted linear layer weights to the second quantization bit width. Then, it traverses each linear layer weight in the first traversal direction. When traversing to the i-th linear layer weight, the linear layer weight of the i-th linear layer weight is updated from the second quantization bit width to the first quantization bit width. At the same time, starting from the j-th linear layer weight, it traverses each linear layer weight along the second traversal direction, and updates the traversed linear layer weights from the second quantization bit width to the third quantization bit width. This continues until the difference between the average bit width of all linear layer weights and the preset average bit width is within a preset range. Then, the traversal along the second traversal direction is terminated, and the result of the i-th traversal is used as a set of quantization data corresponding to the i-th adjustment.

[0015] Wherein, the first traversal direction is from the start to the end of the sorting; the second traversal direction is from the end of the sorting position to the start of the sorting position; the i-th linear layer weight is the first linear layer weight with the second quantization bandwidth in the first traversal direction, and the j-th linear layer weight is the first linear layer weight with the second quantization bandwidth in the second traversal direction.

[0016] Based on this implementation method, the electronic device iterates through the sorted linear layer weights in a certain order, and adjusts the quantization bit width of the linear layer weights until the adjusted quantized data satisfies the difference between the average bit width and the preset average bit width within a preset range, thus obtaining quantized data. Subsequently, the electronic device can select from multiple sets of quantized data the target quantized data that results in minimal accuracy loss and volume compression when quantizing and compressing the model.

[0017] In another possible implementation of the first aspect, the multiple adjustment operations end when the weight of the i-th linear layer is the same as the weight of the j-th linear layer.

[0018] Based on this implementation method, during the process of the electronic device traversing the linear layer weights in the first sorting direction and traversing each linear layer weight along the second traversal direction starting from the j-th linear layer weight, when the electronic device determines that the same linear layer weight has been traversed from both directions simultaneously, the electronic device stops adjusting the quantization bit width of the linear layer weights.

[0019] In another possible implementation of the first aspect, the preset sorting rule is to sort the linear layer weights from largest to smallest according to their target sensitivity; the target sensitivity of the linear layer weights is used to indicate the degree of influence on the model's performance indicators after quantizing the linear layer weights with different quantization bit widths.

[0020] Based on this implementation method, the linear layer weights in the sorted linear layer weights are arranged in descending order of target sensitivity. Since the greater the target sensitivity of a linear layer weight, the greater its influence on the model's output, in order to reduce the problem of excessive compression of linear layer weights with high target sensitivity, which could significantly affect the model's accuracy, the electronic device can set different quantization bit widths for linear layer weights at different sorting positions according to their order in the sorted linear layer weights, thereby obtaining quantized data.

[0021] In another possible implementation of the first aspect, the target sensitivity of the linear layer weights is determined based on a first sensitivity and a second sensitivity, wherein the first sensitivity is a block-level sensitivity and the second sensitivity is a linear layer-level sensitivity.

[0022] In this embodiment, the electronic device combines block-level sensitivity and linear layer-level sensitivity to determine the target sensitivity of each linear layer weight, thereby more comprehensively evaluating the quantification results of the linear layer weights of the model. This allows the target sensitivity of the linear layer weights to accurately measure the importance of the linear layer weights to the model.

[0023] In another possible implementation of the first aspect, the above method may further include:

[0024] For each target block in at least one block of the model, a first sensitivity of the target block is determined based on the similarity between the input and output information of the target block. For each target linear layer weight included in the target block, the ratio between the number of weight values ​​in the weight matrix of the target linear layer weight that are greater than a first preset threshold and the total number of weight values ​​included in the weight matrix of the target linear layer weight is determined as the second sensitivity of the target linear layer weight. The first sensitivity of the target block is normalized to obtain the third sensitivity of the target block. The second sensitivity of each target linear layer weight in at least one target linear layer weight is normalized to obtain the fourth sensitivity of each target linear layer weight. For each target linear layer weight, the fourth sensitivity of the target linear layer weight and the third sensitivity of the target block are summed to obtain the target sensitivity of the target linear layer weight.

[0025] Based on this implementation method, the electronic device first calculates the first sensitivity of each target block, and then calculates the second sensitivity of each target linear layer weight included in the target block. The electronic device then normalizes the sensitivity of the target block and each target linear layer weight included in the target block using the same normalization method. Based on the normalized third sensitivity of the target block and the fourth sensitivity of the target linear layer weights, the target sensitivity of the target linear layer weights is determined. The resulting target sensitivity of the linear layer weights combines block-level sensitivity and linear layer-level sensitivity, more accurately measuring the importance of the linear layer weight to the model.

[0026] In another possible implementation of the first aspect, the first sensitivity of the target block is determined based on the similarity between the input and output information of the target block, including:

[0027] Calculate the cosine similarity between the input and output information of the target block; calculate the difference between the preset value and the cosine similarity to obtain the first sensitivity of the target block.

[0028] Based on this implementation method, the electronic device determines that the higher the cosine similarity between the input information and the output information, the smaller the difference between the output information and the input information after the input information passes through the target block. For the model, the target block is relatively redundant. Therefore, the electronic device can measure the sensitivity of the target block based on the cosine similarity between the input information and the output information of the target block.

[0029] In another possible implementation of the first aspect, for each target linear layer weight included in the target block, the ratio between the number of weight values ​​in the weight matrix of the target linear layer weights that are greater than a first preset threshold and the total number of weight values ​​included in the weight matrix of the target linear layer weights is determined as the second sensitivity of the target linear layer weights, which may include:

[0030] Normalize each weight value in the weight matrix of each target linear layer weight to obtain a normalized weight matrix, which includes the normalized weight values. The ratio between the number of weight values ​​in the normalized weight matrix that are greater than a second preset threshold and the total number of weight values ​​included in the normalized weight matrix is ​​determined as the second sensitivity of the target linear layer weight.

[0031] Based on this implementation method, the electronic device normalizes each weight value in the weight matrix of the target linear layer weights, and then identifies outliers as weight values ​​in the weight matrix of each target linear layer weight that are greater than a second preset threshold. The more outliers there are, the more dispersed the distribution of the target linear layer weights. Therefore, the electronic device can use the proportion of outliers as a second sensitivity of the target linear layer weights to measure their importance to the model.

[0032] In another possible implementation of the first aspect, the above method may further include:

[0033] The weight matrix for calculating the weights of the target linear layer includes the mean and standard deviation of all weight values.

[0034] Normalize each weight value in the weight matrix of each target linear layer weight to obtain the normalized weight matrix of the target linear layer weights, including:

[0035] For each weight value in the weight matrix, calculate the difference between the weight value and the mean. After obtaining the first difference, calculate the ratio between the first difference and the standard deviation to obtain the normalized weight value.

[0036] Based on this implementation method, the electronic device normalizes all weight values ​​included in the weight matrix of the target linear layer weights, so as to more accurately calculate the proportion of outliers in the weight matrix and improve the accuracy of determining the sensitivity of the target linear layer weights.

[0037] In another possible implementation of the first aspect, the above method may further include:

[0038] Calculate the second difference between the first sensitivity and the minimum sensitivity of the target block, and the third difference between the maximum sensitivity and the minimum sensitivity. The minimum sensitivity is the minimum of the first sensitivity of the target block and the second sensitivity of the multiple target linear layer weights, and the maximum sensitivity is the maximum of the first sensitivity of the target block and the second sensitivity of the multiple target linear layer weights.

[0039] The third sensitivity of the target block is obtained by normalizing the first sensitivity of the target block, including:

[0040] Calculate the ratio between the second difference and the third difference to obtain the third sensitivity;

[0041] The second sensitivity of each target linear layer weight in at least one target linear layer weight is normalized to obtain the fourth sensitivity of each target linear layer weight, including:

[0042] Calculate the difference between the second sensitivity and the minimum sensitivity of each target linear layer weight to obtain the fourth difference. Then, calculate the ratio between the fourth difference and the third difference to obtain the fourth sensitivity of the target linear layer weight.

[0043] Based on this implementation method, the target sensitivity of the linear layer weights determined by the electronic device combines block-level sensitivity and linear layer-level sensitivity. After the electronic device determines the sensitivity of the block and the sensitivity of all linear layer weights in the block, it can normalize the block sensitivity and the sensitivity of all linear layer weights in the block together to obtain the normalized block sensitivity and the normalized sensitivity of the linear layer weights, so that the target sensitivity of the linear layer weights can be determined subsequently based on the normalized sensitivity.

[0044] In another possible implementation of the first aspect, the performance metrics include perplexity and / or loss value, and the performance metrics meet preset conditions, including: perplexity less than a preset perplexity threshold and / or loss value less than a preset loss value threshold.

[0045] Based on this implementation method, the electronic device can measure the performance indicators of candidate models based on the perplexity and / or loss values ​​of multiple candidate models. A smaller perplexity value indicates better performance of the candidate model, and a smaller loss value also indicates better performance. Of course, the electronic device can also use other parameters to measure the performance indicators of candidate models, and this embodiment does not limit the scope of this application.

[0046] In a second aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the model quantization method described in any one of the first aspects above.

[0047] Thirdly, this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the model quantization method described in any one of the first aspects.

[0048] Fourthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the model quantization method described in any one of the first aspects.

[0049] It is understood that the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0050] Figure 1 A schematic diagram of the structure of a model system provided in an embodiment of this application;

[0051] Figure 2 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0052] Figure 3 A schematic flowchart of a model quantization method provided in an embodiment of this application;

[0053] Figure 4 A flowchart illustrating a method for determining quantized data of a model provided in an embodiment of this application;

[0054] Figure 5 A schematic diagram illustrating the determination of target sensitivity for the target linear layer weights, provided in an embodiment of this application;

[0055] Figure 6 This is an example diagram illustrating the sorting of thread layer weights provided in an embodiment of this application;

[0056] Figure 7 An example diagram illustrating a process for determining target quantification data provided in an embodiment of this application;

[0057] Figure 8 An example diagram illustrating the application of a quantized model provided in this application embodiment;

[0058] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0060] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0061] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] First, to facilitate understanding of the method provided in this application, the technical terms involved in this application are explained as follows:

[0063] Linear layer: This is a basic layer in the model, also known as a fully connected layer. The main task of a linear layer is to perform a linear transformation on the input data through weights and biases to generate output data.

[0064] Linear layer weights: These are the parameters used in a linear layer to transform input data into output data. Linear layer weights can be a matrix, where each element represents the connection strength between the input and output features. If a linear layer receives an input vector of dimension n and produces an output vector of dimension m, then the weight matrix of that layer is an m*n matrix.

[0065] Quantization compression refers to the process of reducing the parameter data and computational load of a model through a series of techniques, thereby improving the model's running efficiency.

[0066] Quantization bit width: refers to the number of binary bits used to quantize the weights of a linear layer. The lower the quantization bit width, the smaller the model size, and the less memory the electronic device needs to store the linear layer weights.

[0067] In related technologies, electronic devices quantize and compress models by applying the same quantization precision to all linear layer weights within the model. For example, an electronic device can compress all linear layer weights using the same high quantization bit width (e.g., 4 bits). However, since some linear layers have little or no impact on model performance, their weights can be further compressed. Therefore, while electronic devices can compress all linear layer weights using the same high quantization bit width, this approach results in insufficient model compression and a large model size.

[0068] Alternatively, electronic devices can compress all linear layer weights of the model using the same low quantization bit width (e.g., 2 bits). However, because some linear layer weights have outliers that affect quantization, the model suffers significant accuracy loss after extremely low bit width quantization (e.g., 2 bits), limiting further compression of the model size.

[0069] Therefore, this application provides a model quantization method applied to an electronic device. After the electronic device quantizes the linear layer weights of a model based on multiple sets of quantized data to obtain multiple quantized candidate models, it performs performance tests on these candidate models using the same test set to determine the performance index of each candidate model. The quantized data corresponding to the target model whose performance index meets preset conditions among the multiple candidate models is then used as the target quantized data for the model. Specifically, one set of quantized data is used to obtain one candidate model, and another set of quantized data is used to indicate the quantization bit width of the linear layer weights of the model; at least two linear layer weights have different quantization bit widths.

[0070] Because a smaller quantization bit width results in greater precision loss for linear layer weights, using the same low quantization bit width for all linear layer weights leads to significant precision loss in some linear layer weights, thus affecting the overall model accuracy. Conversely, using the same large quantization bit width for all linear layer weights results in insufficient compression of some linear layer weights, leaving room for further compression of the compressed model's size. In this embodiment, the electronic device uses at least two different quantization bit widths for the linear layer weights, solving the problems of severe precision loss when using the same low quantization bit width for all linear layer weights, and insufficient compression and large memory usage when using the same high quantization bit width. Furthermore, the electronic device uses the quantized data of the target model whose performance metrics meet preset conditions as the target quantized data, resulting in less precision loss and smaller size of the quantized target model.

[0071] In this embodiment, the model is a trained model. After the model training is complete and its performance meets preset conditions, the electronic device can quantize and compress the model to reduce its size and reduce the consumption of computing resources. The model may include at least one block, each block includes at least one linear layer, and each linear layer has corresponding linear layer weights.

[0072] The embodiments of this application do not limit the type of model. The model can be a large model such as a large language model, a computer vision model, or a reinforcement learning model, or it can be another type of model with fewer parameters compared to a large model. That is to say, the quantization method of the model in the embodiments of this application is not only applicable to quantizing the parameters of large models, but also applicable to quantizing other models.

[0073] The model quantization method in this application is applied to the server side. The model obtained after compression using the target quantized data can be integrated on the server side or on the terminal device side; this application does not limit this. In other words, the process of the model quantization method provided in this application is executed by the server. Accordingly, all parts (e.g., units) of the model quantization device are all located in the server. After determining the target quantization data for quantizing the model, the server uses the target quantization data to quantize the model, obtaining the quantized model. The server can deploy the quantized model on the server or on the terminal device; this application does not limit this.

[0074] like Figure 1 As shown, Figure 1 This is a schematic diagram of a model system provided in an embodiment of this application. The model system includes a terminal device 101 and a server 102. The terminal device 101 and the server 102 can transmit data through a network 103. The network 103 can include various connection types, such as wired or wireless communication links or fiber optic cables, etc., and this application does not limit it in this regard.

[0075] For example, the electronic device used to perform the method described in this application can be Figure 1 The server 102 is used. The model obtained after target quantization data compression can be integrated into the terminal device 101. The terminal device 101 can be a tablet computer 111, a mobile phone 112, or a desktop computer 113. The terminal device 101 can also be a laptop computer, a personal computer (PC), a smart speaker, or a smartwatch, etc., and this application is not limited to these.

[0076] For example, the electronic device used to perform the method described in this application can be Figure 1 The model obtained after target quantization data compression can also be integrated into server 102. Server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0077] Server 102 can be a server that provides various services. For example, suppose that the model obtained by quantizing the target quantized data is deployed on server 102. Server 102 receives input information about querying the weather sent by the user through mobile phone 112, processes the input information based on the model, generates and feeds back the processing results for the input information about querying the weather to the user.

[0078] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a server structure provided in an embodiment of this application. The server can be a big data platform or a data processing server, or it can be a device integrating a big data platform or a data processing server. The server will be described in detail below. It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the server. In other embodiments, the server may include... Figure 2 It may contain more or fewer components, or combine some components, or separate some components, or arrange the components differently. The components shown in the diagram may be implemented in hardware, software, or a combination of software and hardware.

[0079] like Figure 2 As shown, the server may include a processor 210, a memory 220, and a communication module 230. The processor 210 can be used to read and execute computer-readable instructions. Specifically, the processor 210 may include a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. Registers are high-speed storage components with limited storage capacity, which can be used to temporarily store instructions, data, and addresses.

[0080] The processor 210 may further include a model training module 211 and a model quantization module 212. The model training module 211 can be used to train the model so that the trained model can accurately output prediction results. The model quantization module 212 can be used to quantize and compress the trained model to obtain a smaller model, thereby improving the model's running efficiency.

[0081] In specific implementations, the hardware architecture of processor 210 can be an application-specific integrated circuit (ASIC) architecture, a microprocessor without interlocked piped stages (MIPS) architecture, an ARM (advanced RISC machines) architecture, or a net processor (NP) architecture, etc.

[0082] The memory 220 is coupled to the processor 210 and is used to store various software programs and / or multiple sets of instructions. In this embodiment, the data storage method of the electronic device can be implemented either by integrating it into a processor of the server or by storing it as program code in the server's memory. A processor of the server can then call the code stored in the server's memory to execute the above method. Specifically, the memory 220 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 may store an operating system, such as uCOS, VxWorks, RTLinux, or other embedded operating systems.

[0083] The communication module 230 can be used to establish communication connections between the server and other communication terminals via a network, and to send and receive data via the network. For example, when an electronic device is powered on and connected to the network, the server establishes a connection with the electronic device through the communication module 230 to facilitate the subsequent transmission of user sample data. For example, when the electronic device collects user sample data reported by the user, the server can receive the user sample data reported by the electronic device.

[0084] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the server. In other embodiments, the server may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0085] The model quantization process provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0086] In this embodiment, the server can quantize the linear layer weights of the model based on multiple sets of quantized data to obtain multiple candidate models. Then, the server uses the same test set to perform performance tests on the multiple candidate models. After determining the performance index of each candidate model, the server uses the quantized data corresponding to the target model whose performance index meets the preset conditions as the target quantized data of the model. Furthermore, the server can use the target quantized data to quantize the model to obtain the quantized model.

[0087] For the specific implementation process described above, please refer to [link to documentation / reference]. Figure 3 The introduction process, Figure 3 This is a flowchart illustrating a model quantization method provided in an embodiment of this application, as shown below. Figure 3 As shown, this method can be applied to a server and may include the following steps:

[0088] S301, the server quantizes the linear layer weights of the model based on multiple sets of quantized data to obtain multiple candidate models.

[0089] In this set of quantized data, a candidate model is obtained, and a set of quantized data is used to indicate the quantization bit width of the linear layer weights of the model. At least two linear layer weights in the set of quantized data have different quantization bit widths.

[0090] In this embodiment, after determining multiple sets of quantized data, the server quantizes the model based on these sets of data to obtain multiple quantized candidate models. In other words, the server adjusts the quantization bit width of the model according to the multiple sets of quantized data to obtain multiple candidate models with adjusted bit width. For example, assuming the server determines 10 sets of quantized data, the server adjusts the quantization bit width of the model according to each of the 10 sets of quantized data to obtain 10 quantized candidate models.

[0091] Since the multiple sets of quantized data are not the same, the performance of the candidate models obtained by quantizing the model with different quantized data will be different. Subsequently, the server can determine the target quantized data for quantizing the model from multiple sets of quantized data based on the performance indicators of multiple candidate models. This allows the size of the model after quantization using the target quantized data to be further compressed, and the accuracy loss of the model to be minimized.

[0092] In this embodiment of the application, the process by which the server determines multiple sets of quantized data is described later. Figure 4 The introduction process will not be detailed here.

[0093] It should be explained that since the nonlinear layers in the model do not contain parameters, there is no need to quantize and compress the nonlinear layers of the model.

[0094] S302, the server uses the same test set to perform performance tests on multiple candidate models and determine the performance metrics of each candidate model.

[0095] The performance metrics for candidate models may include perplexity and / or loss value.

[0096] The perplexity value mentioned above is used to evaluate the model's predictive ability on real-world language data, reflecting the magnitude of the model's prediction uncertainty on the test set. The smaller the perplexity value, the more accurate the model's predictions on the test set, and the better the model's performance.

[0097] The loss values ​​mentioned above are used to evaluate the difference between the model's predictions and the actual values. The smaller the loss value, the more accurate the model's predictions on the test set, and the better the model's performance.

[0098] In some embodiments, after the server determines multiple candidate models, it inputs the input information from the same test set into the multiple candidate models, and determines the perplexity of the candidate models based on the output information of each candidate model and the preset output information in the test set.

[0099] For example, the server can use the following formula (1) to calculate the perplexity of the candidate model.

[0100]

[0101] Where P represents the perplexity of the candidate model, P(x i ) represents the probability of the i-th word predicted by the candidate model, and N represents the total number of words in the test set.

[0102] In other embodiments, after determining multiple candidate models, the server inputs the input information from the same test set into each candidate model, and determines the loss value of each candidate model based on the output information of each candidate model and the preset output information in the test set. For example, the server can use the following formula (2) to calculate the loss value of the candidate model.

[0103]

[0104] Where LOSS represents the loss value of the candidate model, N represents the number of input information items in the test set, and y j This represents the j-th output information of the candidate model. This represents the j-th preset output information in the test set.

[0105] The performance metrics of the candidate models mentioned above may include perplexity and / or loss value as examples only. The server may also use other parameters to measure the performance metrics of the candidate models, which are not limited in this embodiment.

[0106] S303: The server uses the quantized data of the target model whose performance indicators meet the preset conditions from among multiple candidate models as the target quantized data.

[0107] The preset conditions include that the perplexity is less than a preset perplexity threshold, and / or that the loss value is less than a preset loss value threshold.

[0108] In some embodiments, after the server determines the perplexity and / or loss value of multiple candidate models, the server can determine the quantized data of the target model whose perplexity is less than a preset perplexity threshold and / or whose loss value is less than a preset loss value threshold as the target quantized data.

[0109] The aforementioned preset perplexity threshold and preset loss threshold can be set according to the application scenario of the model and the accuracy requirements of the model. This embodiment does not limit this.

[0110] For example, if the perplexity of a candidate model is 0.35 and the preset perplexity threshold is 0.4, and the perplexity of the candidate model is less than the preset perplexity threshold, the server can determine that the quantized data of the candidate model is the target quantized data.

[0111] After determining the performance metrics of multiple candidate models, the server can identify the quantized data of the target model whose performance metrics meet preset conditions as the target quantized data. It is evident that the server quantizes all linear layer weights of the model together and considers the differences in performance metrics on the test set caused by multiple sets of quantized data. This not only allows for the rapid identification of the target quantized data that satisfies the preset conditions but also enables a significant improvement in model accuracy with only a slight increase in size when using the model quantized from the target quantized data for prediction.

[0112] S304, the server uses the target quantization data to quantize the model, and obtains the quantized model.

[0113] In this embodiment of the application, the server adjusts the corresponding linear layer weights in the model according to the quantization bit width corresponding to the linear layer weights of the model indicated by the target quantization data, so as to obtain the quantized model.

[0114] For example, suppose the model includes 7 linear layer weights, and the target quantization data determined by the server includes 4 bits, 4 bits, 3 bits, 3 bits, 3 bits, 2 bits, and 2 bits. The server can quantize these 7 linear layer weights sequentially into 4 bits, 4 bits, 3 bits, 3 bits, 3 bits, 2 bits, and 2 bits.

[0115] Since the target quantization data determined by the server includes multiple quantization bit widths, using this target quantization data to quantize the model results in a quantized model with advantages such as higher accuracy and lower perplexity.

[0116] For example, Figure 4 A flowchart illustrating a method for determining quantization data of a model provided in this application is shown below. Figure 4 As shown, this method can be applied to a server and may include the following steps:

[0117] S401, the server determines the target sensitivity of the weights for each linear layer in the model.

[0118] The target sensitivity of the linear layer weights indicates the degree of influence on the model's output after quantizing the linear layer weights with different quantization bit widths. The higher the target sensitivity of the linear layer weights, the greater their influence on the model's output; conversely, the lower the target sensitivity, the smaller their influence.

[0119] In some embodiments, the server can determine the target sensitivity of a linear layer weight based on the sensitivity of the block to which the linear layer belongs and the sensitivity of the linear layer weight. Thus, the server combines block-level sensitivity and linear layer-level sensitivity to determine the target sensitivity of each linear layer weight, providing a more comprehensive evaluation of the model's linear layer weight quantification results. This allows the target sensitivity of a linear layer weight to accurately measure its importance to the model.

[0120] In some embodiments, the server may first determine the sensitivity of each block and the sensitivity of each linear layer weight. Then, for each target block, the server may normalize the first sensitivity of the target block and the second sensitivity of at least one target linear layer weight included in the target block together to obtain the normalized sensitivity of the target block (i.e., the third sensitivity) and the fourth sensitivity of at least one target linear layer weight (i.e., the normalized target linear layer weight). Further, for each target linear layer weight, the server sums the fourth sensitivity of the target linear layer weight with the third sensitivity of the target block, and the resulting sensitivity is used as the target sensitivity of the target linear layer weight.

[0121] For example, suppose the model consists of 32 blocks, each block contains 7 linear layers, and each linear layer has corresponding linear layer weights. Figure 5 As shown, for target block 1 and its seven target linear layers, the server first calculates the first sensitivity of target block 1 and the second sensitivity of the weights of the seven target linear layers. Then, it normalizes both the first sensitivity of target block 1 and the second sensitivity of the weights of the seven target linear layers to obtain the third sensitivity of target block 1 and the fourth sensitivity of the weights of the seven target linear layers. For each target linear layer weight, the server calculates the sum of the fourth sensitivity of the target linear layer weight and the third sensitivity of target block 1 to obtain the target sensitivity of the target linear layer weight. It is evident that the server combines block-level sensitivity and linear layer-level sensitivity to more accurately determine the target sensitivity of each linear layer weight, more comprehensively evaluate the linear layer weight quantization results of the model, and more effectively optimize the model's weight quantization, thereby improving inference efficiency and hardware friendliness while maintaining model performance.

[0122] When the server determines the sensitivity of each block in the model, it can determine the first sensitivity of the target block based on the similarity between the input and output information of the target block.

[0123] In some embodiments, the server can calculate the cosine similarity between the input and output information of the target block, and then calculate the difference between a preset value and the cosine similarity to obtain the first sensitivity of the target block. For example, the server can use the following formula (3) to calculate the first sensitivity of each target block.

[0124] BlockScore(Block)=1-CosineSimilarity(I,Block(I)) Formula (3);

[0125] Where BlockScore(Block) represents the first sensitivity of the target block, I represents the input information of the target block, Block(I) represents the output information of the target block, and CosineSimilarity(I, Block(I)) represents the cosine similarity between the input information and the output information.

[0126] It's understandable that the higher the cosine similarity between the input and output information of a target block, the smaller the difference between the output and input information after the input information passes through the target block. For the model, all linear layers within a block with high cosine similarity are relatively redundant, allowing the server to compress all linear layers within that target block to a greater extent. In other words, for target blocks with high cosine similarity, the server can use low-precision quantization with a high compression ratio to explicitly reduce the size of the block and thus reduce the model size.

[0127] In the above formula (3), the purpose of the server calculating the difference between 1 and cosine similarity is to normalize the cosine similarity of all blocks so that the sensitivity of all blocks is within a certain range, which is convenient for subsequent analysis.

[0128] The method described above for the server to determine the first sensitivity of a target block based on the cosine similarity between the input and output information is merely an example. This application does not limit the method by which the server determines the similarity between the input and output information of the target block. For example, the server can determine the similarity between the input and output information based on Euclidean distance, Manhattan distance, etc., and thus determine the first sensitivity of the target block.

[0129] When the server determines the second sensitivity of each target linear layer weight included in the target block, the server can determine the second sensitivity of the target linear layer weight as the ratio between the number of weight values ​​in the weight matrix of the target linear layer weight that are greater than the first preset threshold and the total number of weight values ​​included in the weight matrix of the target linear layer weight.

[0130] In other words, for each target linear layer weight, the server can identify outliers by counting weight values ​​in the weight matrix that exceed a first preset threshold. The more outliers, the more dispersed the distribution of the target linear layer weights, allowing the server to compress the target linear layer to a lesser extent. This means the server can use higher-precision quantized data to quantize and compress the target linear layer weights, avoiding the problem of reduced model accuracy due to a small bit width in the quantization of the target linear layer weights.

[0131] In this embodiment, the server may first normalize each weight value in the weight matrix of the target linear layer weights to obtain a weight matrix including the normalized weight values. Then, the server determines the second sensitivity of the target linear layer weights as the ratio between the number of weight values ​​in the normalized weight matrix that are greater than a second preset threshold and the total number of weight values ​​included in the normalized weight matrix.

[0132] In some embodiments, the server may first calculate the mean and standard deviation of all weight values ​​included in the weight matrix of multiple target linear layer weights in the target block. Then, for each weight value in the weight matrix, the server calculates the difference between the weight value and the mean, obtains the first difference, and then calculates the ratio between the first difference and the standard deviation to obtain the normalized weight value. For example, the server may use the following formula (4) to calculate the normalized weight value.

[0133]

[0134] Where m is a positive integer, z m ω represents the weight value of the m-th target linear layer after normalization. m denoted as the weight value of the m-th target linear layer, μ represents the mean of all weight values ​​included in the weight matrix of the same target linear layer, and σ represents the standard deviation of all weight values ​​included in the weight matrix of the same target linear layer.

[0135] For example, assuming the second preset threshold is 3 times the standard deviation, for each normalized weight value, the server can calculate the ratio between the number of weight values ​​outside the 3-standard-deviation threshold and the total number of weight values ​​included in the weight matrix of the target linear layer weights, and determine the second sensitivity of the target linear layer weights. The server can use the following formula (5) to calculate the second sensitivity of the target linear layer weights.

[0136]

[0137] Where n is a positive integer, z nσ represents the weight value of the nth target linear layer weight after normalization, σ represents the standard deviation of all weight values ​​included in the weight matrix of the same target linear layer weight, and N represents the total number of weight values ​​included in the weight matrix of the target linear layer weight.

[0138] Since the sensitivity of a block and the sensitivity of linear layer weights differ significantly in value, to ensure that the values ​​of both sensitivity and sensitivity of the block and linear layer weights fall within a certain range for easier subsequent model analysis, the server determines the first sensitivity of the target block and the second sensitivity of all target linear layer weights within that block. Then, it normalizes these values ​​to obtain a normalized third sensitivity of the target block and a fourth sensitivity of the target linear layer weights. In some embodiments, after determining the first sensitivity of the target block and the second sensitivity of all target linear layer weights within that block, the server can determine a minimum sensitivity and a maximum sensitivity from these values. The minimum sensitivity is the minimum value among the first sensitivity of the target block and the second sensitivity of the multiple target linear layer weights, and the maximum sensitivity is the maximum value among the first sensitivity of the target block and the second sensitivity of the multiple target linear layer weights.

[0139] Then, the server can calculate a second difference between the first sensitivity and the minimum sensitivity of the target block, and a third difference between the maximum sensitivity and the minimum sensitivity. The server then calculates the ratio between the second and third differences to obtain the third sensitivity of the target block. The server can also calculate the difference between the second and minimum sensitivity of each target linear layer weight, obtaining a fourth difference, and then calculate the ratio between the fourth and third differences to obtain the fourth sensitivity of the target linear layer weights.

[0140] For example, the server can use the following formula (6) to normalize the first sensitivity of the target block and the weights of all target linear layers in the target block, so as to obtain the third sensitivity of the target block and the fourth sensitivity of the target linear layer weights after normalization.

[0141]

[0142] in, S represents the third sensitivity of the normalized target block or the fourth sensitivity of the target linear layer weights, S represents the first sensitivity of the target block or the second sensitivity of each target linear layer weight in the target block, min(S) represents the minimum sensitivity, and max(S) represents the maximum sensitivity.

[0143] After the server calculates the third sensitivity of the normalized target block and the fourth sensitivity of the target linear layer weight according to the above formula (6), it can calculate the target sensitivity of the target linear layer weight according to the following formula (7).

[0144]

[0145] Where k is a positive integer, and T_S(k) represents the target sensitivity of the k-th target linear layer weight. This represents the fourth sensitivity of the normalized weights of the k-th target linear layer. This represents the third sensitivity of the normalized target block.

[0146] In the above embodiments, the server takes the calculation of the target sensitivity of the target linear layer weights included in the target block as an example. The above method of calculating the target sensitivity of the target linear layer weights is applicable to the target sensitivity calculation method of the linear layer weights of all blocks included in the model, and will not be repeated in this embodiment.

[0147] In some embodiments, the server can directly perform multiple adjustment operations on the initial quantization bit width of the linear layer weights of the model without sorting the linear layer weights, thereby obtaining multiple sets of quantized data. Each adjustment operation yields one set of quantized data. Each adjustment operation includes adjusting the quantization bit width of the first portion of the linear layer weights to a first quantization bit width, the quantization bit width of the second portion of the linear layer weights to a second quantization bit width, and the quantization bit width of the third portion of the linear layer weights to a third quantization bit width, until the difference between the average bit width of all adjusted linear layer weights and a preset average bit width is within a preset range.

[0148] It is understandable that the server obtains multiple sets of quantized data, including three quantization bit widths, by adjusting the initial quantization bit width of the linear layer weights multiple times. Subsequently, the target quantized data is determined from the multiple sets of quantized data to quantize the model, thus avoiding the problem of reduced model accuracy caused by using the same quantization bit width to process the model.

[0149] In other embodiments, the server can sort the linear layer weights in the model according to a preset sorting rule and set the initial quantization bit width corresponding to the sorted linear layer weights as the second quantization bit width. Then, the server performs multiple adjustment operations on the initial quantization bit width of the linear layer weights of the model to obtain multiple sets of quantized data. When the preset sorting rule is to sort the linear layer weights from largest to smallest according to their target sensitivity, the process of the server determining multiple sets of quantized data can be found in S402 and S403 below.

[0150] Optionally, in S402, the server sorts all linear layer weights in descending order of target sensitivity to obtain the sorted linear layer weights.

[0151] In this embodiment, after the server determines the target sensitivity of all linear layer weights in the model, it sorts all linear layer weights in descending order of target sensitivity to obtain the sorted linear layer weights.

[0152] For example, such as Figure 6 As shown in (a), assuming the initial model includes 224 linear layer weights, after the server determines the target sensitivity of these 224 linear layer weights, it sorts the 224 linear layer weights in descending order of target sensitivity, resulting in... Figure 6 The sorted linear layer weights are shown in (b).

[0153] It's understandable that the sorted linear layer weights are arranged in descending order of target sensitivity. Since the greater the target sensitivity of a linear layer weight, the greater its impact on the model's output, using a smaller quantization bit width to compress these weights would significantly reduce the model's accuracy. Therefore, to mitigate the problem of excessive compression affecting model accuracy, the server can set different quantization bit widths for linear layer weights with different target sensitivities based on their order in the sorted weights, obtaining quantized data. The server then uses this adjusted quantization bit width to quantize the model, resulting in a model with a relatively smaller file size and higher accuracy.

[0154] S403, the server performs multiple adjustment operations on the initial quantization bit width in the sorted linear layer weights to obtain multiple sets of quantized data.

[0155] For example, if a server converts a linear layer weight from a 32-bit floating-point number to an 8-bit integer, the quantization bit width of that linear layer weight is 8 bits, and the weight can be reduced to one-quarter of its original size. A larger quantization bit width results in greater precision loss for the linear layer weights. Therefore, the server can set different quantization bit widths for the linear layer weights ranked earlier and later, ensuring that the difference between the average bit width of all linear layer weights and a preset average bit width remains within a predetermined range.

[0156] In some embodiments, after the server sets the initial quantization bit width of all linear layer weights in the sorted linear layer weights to the second quantization bit width, the server starts to traverse each linear layer weight in the first traversal direction, and adjusts the quantization bit width of the linear layer weights traversed from the first traversal direction from the second quantization bit width to the first quantization bit width. When traversing to a linear layer weight in the first traversal direction, each linear layer weight is traversed starting from the j-th linear layer weight in the second traversal direction. For each linear layer weight traversed along the second traversal direction, the quantization bit width of that weight is adjusted from the second quantization bit width to the third quantization bit width. The average bit width of all linear layer weights at this point is calculated. If the difference between the average bit width and the preset average bit width is within a preset range, the traversal in the second traversal direction stops. Otherwise, if the difference is not within the preset range, the traversal continues to the next linear layer weight along the second traversal direction, adjusting the quantization bit width of each traversed linear layer weight, until the difference between the average bit width of all linear layer weights and the preset average bit width is within the preset range, at which point the traversal in the second traversal direction stops. After stopping the traversal in the second traversal direction, the traversal continues to the next linear layer weight along the first traversal direction. This process is repeated until the weight of the linear layer traversed along the first traversal direction is the same as the weight of the j-th linear layer traversed along the second traversal direction. At this point, the traversal in the first traversal direction is stopped, which means the process of adjusting the linear layer weights is stopped.

[0157] In this process, a single traversal along the first traversal direction is considered an adjustment operation, and the result of a single traversal along the first traversal direction is considered a set of quantized data.

[0158] For example, taking the iteration to the i-th linear layer weight as an example, when iterating to the i-th linear layer weight, the quantization bit width of the i-th linear layer weight is updated from the second quantization bit width to the first quantization bit width. Simultaneously, starting from the j-th linear layer weight, each linear layer weight is iterated along the second traversal direction, and the traversed linear layer weights are updated from the second quantization bit width to the third quantization bit width, until the difference between the average bit width of all linear layer weights and the preset average bit width is within a preset range. Then, the iteration along the second traversal direction is terminated, and the result of the i-th iteration is used as the set of quantization data corresponding to the i-th adjustment. Next, the (i+1)-th linear layer weight is iterated, and the result of the (i+1)-th iteration is used as the set of quantization data corresponding to the (i+1)-th adjustment.

[0159] Wherein, the first traversal direction is from the start to the end of the sorting; the second traversal direction is from the end of the sorting position to the start of the sorting position; the i-th linear layer weight is the first linear layer weight with the second quantization bandwidth in the first traversal direction, and the j-th linear layer weight is the first linear layer weight with the second quantization bandwidth in the second traversal direction.

[0160] For example, the server can construct a first pointer and a second pointer. The first pointer initially points to the linear layer weight that is first in the sorted order, and the second pointer initially points to the linear layer weight that is last in the sorted order. The linear layer weight that the first pointer traverses is the first quantization bit width, and the linear layer weight that the second pointer traverses is the third quantization bit width. The first quantization bit width is greater than the second quantization bit width, and the second quantization bit width is greater than the third quantization bit width.

[0161] In other words, after sorting, the initial quantization width of all linear layer weights is set to the second quantization width. When the first pointer traverses to a certain thread layer weight, the server updates all linear layer weights traversed by the first pointer from the second quantization width to the first quantization width. When the second pointer traverses to a certain linear layer weight, the server updates all linear layer weights traversed by the second pointer from the second quantization width to the third quantization width.

[0162] In some embodiments, when the first pointer traverses to the i-th linear layer weight in the first traversal direction, the server updates the linear layer weight of the i-th linear layer weight from the second quantization bit width to the first quantization bit width. Simultaneously, the server controls the second pointer to traverse the linear layer weights in the second traversal direction, updating the traversed linear layer weights from the second quantization bit width to the third quantization bit width, until the difference between the average bit width of all linear layer weights and the preset average bit width is within a preset range. The traversal along the second traversal direction is then terminated, and the result of the i-th traversal is used as the set of quantized data corresponding to the i-th adjustment. Here, i is a positive integer greater than 0, and the preset average bit width is greater than the third quantization bit width and less than the first quantization bit width.

[0163] When the first pointer initially points to the first linear layer weight in the sorted order, the quantization bit width of the first linear layer weight is adjusted from the second quantization bit width to the first quantization bit width. The server controls the second pointer to traverse the linear layer weights in the second traversal direction, adjusting the quantization bit width of all linear layer weights traversed by the second pointer from the second quantization bit width to the third quantization bit width, until the difference between the average bit width of all linear layer weights and the preset average bit width is within the preset range, and then the traversal along the second traversal direction is terminated, obtaining the initial quantized data (i.e., the first set of quantized data) corresponding to all linear layer weights.

[0164] For example, such as Figure 7As shown in (a), assuming the server sets the initial quantization bit width of all linear layer weights in the sorted linear layer weights to 3 bits, and the preset average bit width is 2.5 bits, pointer J initially points to the first linear layer weight 2, and pointer K initially points to the last linear layer weight 223. At this time, the quantization bit width of linear layer weight 1 is 4 bits, and the quantization bit width of linear layer weight 223 is 2 bits. The server controls pointer K to traverse the sorted linear layer weights from back to front, and the quantization bit width of all linear layer weights traversed by pointer K is 2 bits, until the difference between the average quantization bit width of all linear layer weights and the preset average bit width is within a preset range, then pointer K stops moving. Figure 7 As shown in (b), the initial quantization data corresponding to all linear layer weights are obtained.

[0165] The server moves the first pointer in the first traversal direction, shifting it one position to the right. Then, it controls the second pointer to continue traversing the linear layer weights in the second traversal direction until the difference between the average quantization bit width of all linear layer weights and the preset average bit width is within a preset range. At this point, the server stops moving the second pointer, obtaining the second set of quantized data. This process is repeated, with the server controlling the movement of the first and second pointers. When both pointers point to the same linear layer weight, the server stops traversing the linear layer weights, obtaining multiple sets of quantized data. Subsequently, the server can select target quantized data from these multiple sets that allows the candidate model's performance metrics to meet preset conditions for model quantization.

[0166] For example, such as Figure 7 As shown in (c), when the server control pointer J and pointer K point to the same linear layer weight, the server stops adjusting the linear layer weight. Subsequently, the target quantized data determined by the server from multiple sets of quantized data can be processed as follows: Figure 7 As shown in (d).

[0167] In this embodiment, the multiple sets of quantized data determined by the server include at least one first quantization bit width, at least one second quantization bit width, and at least one third quantization bit width. It can be seen that the server uses quantization data of three precision levels to quantize the initial model. Compared with related technologies that use only one precision level of quantization data to quantize the model, this avoids the problem of excessive quantization amplitude for the weights of highly sensitive linear layers, which would have a significant impact on the accuracy of the model. This allows the server to determine the target quantization data from multiple sets of quantization data and then use the target quantization data to quantize the model, resulting in a more accurate quantized model.

[0168] In summary, in this embodiment, when determining the target sensitivity of the linear layer weights of the model, the server combines block-level sensitivity and linear layer-level sensitivity to more comprehensively evaluate the impact of linear layer weight quantization on model performance and more effectively optimize model weight quantization. Therefore, the multiple sets of quantization data determined by the server include various quantization bit widths. After determining the target quantization data from these multiple sets, the target quantization data is used to quantize the model, resulting in a quantized model with advantages such as higher accuracy and lower perplexity.

[0169] For example, Table 1 below shows the model size, accuracy, and perplexity after the server quantizes the model using only one level of quantization data. Table 1 shows that when the server determines the original quantization bit width of all linear layer weights of the model to be 16 bits, the model size is 4.98 GB. When the server uses 4 bits to quantize all linear layer weights of the model, the model size is 2.05 GB. When the server uses 2 bits to quantize all linear layer weights of the initial model, the model size is 1.56 GB. It can be seen that as the quantization bit width used by the server to quantize all linear layer weights of the model decreases, the model size decreases. However, when the server uses the same test set to test the performance metrics of the quantized model, it is found that the model's accuracy decreases as the quantization bit width decreases, and the model's perplexity increases as the quantization bit width decreases. In other words, the model's performance decreases as the quantization bit width decreases.

[0170] As shown in Table 2, when the server uses 2.5 bits to quantize the model, the model size is 1.68 GB. When the server uses 3 bits to quantize the model, the model size is 1.8 GB. Compared to Table 1, in this embodiment, the server uses multiple precision levels of quantization bit widths to quantize the model, resulting in a significant improvement in model accuracy and a significant reduction in model perplexity.

[0171] For example, as shown in Tables 1 and 2, the server uses multiple precision levels of quantization bit widths with an average bit count of 2.5 bits and 3 bits to quantize the model. Compared with quantizing the model using a single precision level (e.g., 16 bits, 4 bits, or 2 bits), the accuracy of the quantized model is significantly improved, and the perplexity of the model is also significantly reduced.

[0172] The following test sets are listed in Tables 1 and 2: Massive Multitask Language Understanding (MMLU), C-Eval, Contrastive Contextual Corruption Corpus (C3), Physical Interaction Question Answering (PIQA), Abstraction and Reasoning Corpus-Easy (ARC-E), Abstraction and Reasoning Corpus-Challenge (ARC-C), Colossal Clean Crawled Corpus (C4), and Wikitext2. Of course, the server can also use other test sets to test the quantized model, such as Hellaswag or Winogrande. The type of test set is not limited in this embodiment.

[0173] Table 1

[0174]

[0175] In this embodiment, the server uses target quantization data to quantize the model. After obtaining the quantized model, the quantized model can be deployed on the server or on the terminal device, which is not limited here.

[0176] Taking the deployment of the quantized model on a server as an example, the application with dialogue capabilities on the terminal device (hereinafter referred to as the target application) receives user input information and transmits it to the server. After receiving the user input information, the server performs inference based on the quantized model (e.g., a large language model) to obtain the inference result.

[0177] Taking a large language model as an example, the server trains the large language model to obtain the trained large language model. Then, after determining the target quantization data to be used for quantization processing of the large language model, the server uses the target quantization data to quantize the large language model to obtain the quantized target large language model.

[0178] In some embodiments, when a user interacts with a target application, the user can input information into the application via voice interaction. After receiving the voice information, the target application can recognize the voice information as text information and send it to the server. Furthermore, the user input information received by the terminal device can also be input by the user through gesture operations or virtual keyboard input operations; the method of user input is not limited here.

[0179] In one scenario, the server can send the inference result to the target application. Upon receiving the inference result, the target application determines the dialogue outcome based on the inference and then presents the dialogue outcome to the user or plays it via voice playback. The target application can be an application with dialogue functionality, such as a voice assistant or a third-party interactive application.

[0180] In another scenario, the server can determine the dialogue outcome based on the reasoning result, and then send the dialogue outcome to the target application so that the target application can present the dialogue outcome to the user or play the dialogue outcome via voice playback.

[0181] For example, taking a mobile phone as the terminal device and a voice assistant as the target application, a user can wake up the voice assistant by voice or by long-pressing the power button. After the voice assistant is woken up, the phone can display the voice assistant's interface. The user can then access the smart interaction interface through the voice assistant's interface. For instance, after the phone receives the user's voice message "Xiao Y Xiao Y" to wake up the voice assistant, it displays... Figure 8 Interface 800 is shown in (a). Alternatively, after the phone receives a long press operation from the user on the power button 801, it responds to the long press operation by displaying... Figure 8 Interface 800 is shown in (a). Then, after the phone receives the user's voice message "Check the weather in city A tomorrow," the phone can recognize the user's voice message and display the recognized message, such as... Figure 8 The voice message recognized by the phone and displayed in interface 802 shown in (b) is "Please check the weather in city A tomorrow." After the phone confirms that the user has completed the voice message input, it can display... Figure 8 The interactive interface 803 shown in (c) includes an interface provided by the voice assistant that displays the dialogue results.

[0182] In some embodiments, assuming the target large language model is deployed on the server side, after the mobile phone determines the user input information based on the recognized voice information, it sends the user input information to the server. After receiving the user input information, the server performs inference based on the target large language model and sends the inference result to the mobile phone. After receiving the inference result sent by the server, the mobile phone searches for the dialogue result based on the inference result and presents the dialogue result as "Okay, the weather in city A will be sunny tomorrow" in the voice assistant's interactive interface 803.

[0183] The above description uses the target application as a voice assistant and weather prediction as an example to illustrate the function of the target large language model. The applications that interact with the user via voice in the terminal device are not limited in this embodiment. Furthermore, the above description uses the trained and quantized target large language model integrated on the server side as an example. In this embodiment, when the trained and quantized target large language model is integrated on the terminal device, after the terminal device obtains the user's input information, it does not need to interact with the server side. The terminal device directly performs inference based on the target large language model, obtains the inference result, and then presents the dialogue result obtained based on the inference result to the user or plays it via voice playback.

[0184] like Figure 9 As shown in the figure, this application discloses an electronic device. Specifically, the electronic device may include: one or more processors 901; a memory 902; one or more application programs (...). Figure 9 (not shown); and one or more computer programs 903, wherein the aforementioned devices can be connected via one or more communication buses 904. The one or more computer programs 903 are stored in the aforementioned memory 902 and configured to be executed by the one or more processors 901, and the one or more computer programs 904 include instructions that can be used to perform the relevant steps in the above embodiments.

[0185] It is understood that the aforementioned electronic devices, etc., include hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this invention.

[0186] This application embodiment can divide the above-mentioned electronic device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. The module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0187] When each functional module is divided according to its corresponding function, the above embodiments illustrate a possible composition of the electronic device, which may include a display unit, a transmission unit, and a processing unit. All relevant content regarding the steps in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0188] Embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform the aforementioned method steps to implement the model quantization method in the above embodiments.

[0189] Embodiments of this application also provide a computer program product, which includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the aforementioned method steps to implement the model quantization method in the above embodiments.

[0190] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to enable the apparatus to perform the model quantization method executed by the electronic device in the above-described method embodiments.

[0191] In this embodiment, the electronic device, computer-readable storage medium, computer program product or device are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0192] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0193] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0195] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model quantization method, characterized in that, The method includes: The linear layer weights in the model are sorted according to the target sensitivity ranking rule from largest to smallest to obtain the sorted linear layer weights. The target sensitivity of the linear layer weights is determined based on a first sensitivity and a second sensitivity, where the first sensitivity is a block-level sensitivity and the second sensitivity is a linear layer-level sensitivity. For each target block in at least one block of the model, the first sensitivity of the target block is determined based on the similarity between the input and output information of the target block. For each target linear layer weight included in the target block, the second sensitivity of the target linear layer weight is determined by the ratio between the number of weight values ​​in the weight matrix of the target linear layer weight that are greater than a first preset threshold and the total number of weight values ​​included in the weight matrix of the target linear layer weight. Multiple adjustment operations are performed on the initial quantization bit width of the sorted linear layer weights to obtain multiple sets of quantized data. Each adjustment operation yields one set of quantized data. Each adjustment operation includes adjusting the quantization bit width of the first portion of the linear layer weights to a first quantization bit width, the quantization bit width of the second portion of the linear layer weights to a second quantization bit width, and the quantization bit width of the third portion of the linear layer weights to a third quantization bit width, until the difference between the average bit width of all adjusted linear layer weights and a preset average bit width is within a preset range. The first quantization bit width is greater than the second quantization bit width, and the second quantization bit width is greater than the third quantization bit width. The linear layer weights of the model are quantized based on the multiple sets of quantized data to obtain multiple candidate models. The model is a large language model with text as input. One set of quantized data is used to obtain a candidate model, and another set of quantized data is used to indicate the quantization bit width of the linear layer weights of the model. At least two linear layer weights have different quantization bit widths. The performance of the multiple candidate models is tested using the same test set to determine the performance metrics of each candidate model. Among multiple candidate models, the quantitative data corresponding to the target model whose performance indicators meet the preset conditions is used as the target quantitative data of the model. The model is quantized using the target quantization data to obtain a quantized model; the quantized model is deployed on a terminal device and implemented through the processor of the terminal device.

2. The method according to claim 1, characterized in that, The method further includes: Set the initial quantization bit width corresponding to the sorted linear layer weights to the second quantization bit width; Starting from the first traversal direction, each linear layer weight is traversed. When traversing to the i-th linear layer weight, the linear layer weight of the i-th linear layer weight is updated from the second quantization bit width to the first quantization bit width. At the same time, starting from the j-th linear layer weight, each linear layer weight is traversed along the second traversal direction, and the traversed linear layer weights are updated from the second quantization bit width to the third quantization bit width. This continues until the difference between the average bit width of all linear layer weights and the preset average bit width is within the preset range. Then, the traversal along the second traversal direction is terminated, and the result of the i-th traversal is used as a set of quantization data corresponding to the i-th adjustment. Wherein, the first traversal direction is from the start to the end of the sorting; the second traversal direction is from the end of the sorting position to the start of the sorting position; the i-th linear layer weight is the first linear layer weight with the second quantization bandwidth in the first traversal direction, and the j-th linear layer weight is the first linear layer weight with the second quantization bandwidth in the second traversal direction.

3. The method according to claim 2, characterized in that, The multiple adjustment operations end when the weight of the i-th linear layer is the same as the weight of the j-th linear layer.

4. The method according to any one of claims 1-3, characterized in that, The target sensitivity of the linear layer weights is used to indicate the degree of impact on the performance index of the model after quantizing the linear layer weights with different quantization bit widths.

5. The method according to claim 1, characterized in that, The method includes: The third sensitivity of the target block is obtained by normalizing the first sensitivity of the target block; The second sensitivity of each target linear layer weight in the at least one target linear layer weight is normalized to obtain the fourth sensitivity of each target linear layer weight; For each target linear layer weight, the fourth sensitivity of the target linear layer weight is summed with the third sensitivity of the target block to obtain the target sensitivity of the target linear layer weight.

6. The method according to claim 5, characterized in that, Determining the first sensitivity of the target block based on the similarity between the input and output information of the target block includes: Calculate the cosine similarity between the input and output information of the target block; The difference between the preset value and the cosine similarity is calculated to obtain the first sensitivity of the target block.

7. The method according to any one of claims 1-3, characterized in that, The second sensitivity of the target linear layer weight is determined by the ratio between the number of weight values ​​greater than a first preset threshold in the weight matrix of the target linear layer weight and the total number of weight values ​​included in the weight matrix of the target linear layer weight, for each target linear layer weight included in the target block. This includes: Normalize each weight value in the weight matrix of each target linear layer weight to obtain a normalized weight matrix, wherein the weight matrix includes the normalized weight values. The ratio between the number of weight values ​​in the normalized weight matrix that are greater than the second preset threshold and the total number of weight values ​​included in the normalized weight matrix is ​​determined as the second sensitivity of the target linear layer weights.

8. The method according to claim 7, characterized in that, The method further includes: Calculate the mean and standard deviation of all weight values ​​included in the weight matrix of the target linear layer weights; The normalization process for each weight value in the weight matrix of each target linear layer weight, to obtain the normalized weight matrix of the target linear layer weight, includes: For each weight value in the weight matrix, the difference between the weight value and the mean is calculated to obtain a first difference; Calculate the ratio between the first difference and the standard deviation to obtain the normalized weight value.

9. The method according to claim 5 or 6, characterized in that, The method further includes: Calculate a second difference between the first sensitivity and the minimum sensitivity of the target block, and a third difference between the maximum sensitivity and the minimum sensitivity, wherein the minimum sensitivity is the minimum of the first sensitivity of the target block and the second sensitivity of the plurality of target linear layer weights, and the maximum sensitivity is the maximum of the first sensitivity of the target block and the second sensitivity of the plurality of target linear layer weights; The process of normalizing the first sensitivity of the target block to obtain the third sensitivity of the target block includes: Calculate the ratio between the second difference and the third difference to obtain the third sensitivity; The normalization process for the second sensitivity of each target linear layer weight in the at least one target linear layer weight to obtain the fourth sensitivity of each target linear layer weight includes: The difference between the second sensitivity of each target linear layer weight and the minimum sensitivity is calculated to obtain the fourth difference; The ratio between the fourth difference and the third difference is calculated to obtain the fourth sensitivity of the target linear layer weight.

10. The method according to any one of claims 1-3, characterized in that, The performance metrics include perplexity and / or loss value, and the performance metrics meet preset conditions, including: The perplexity is less than a preset perplexity threshold and / or the loss value is less than a preset loss value threshold.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-10.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-10.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-10.

Images