Model quantization method, apparatus, device, and storage medium

Abstract

A model quantization method, apparatus, device, and storage medium are disclosed. The method involves inputting acquired historical lexical units into a large language model to be quantized, obtaining a set of candidate lexical units and their probability distributions; determining a first contribution value for each candidate lexical unit based on the feature space and depth-sensing weights of each linear layer in the large language model; determining a target lexical unit from the set of candidate lexical units based on the first contribution value; iteratively executing the operation of inputting the acquired historical lexical units into the large language model to be quantized until the iteration stop condition is met, generating calibration samples; using the calibration samples, quantizing the large language model to be quantized to obtain the quantized target model, which is then deployed to a hardware device. The quantized target model can be called by the hardware device to perform corresponding tasks, realizing the generation of calibration data samples from the geometric perspective of the latent manifold of the large language model to be quantized.

Description

Technical Field

[0001] This disclosure relates to fields such as artificial intelligence technology and computer vision technology, and in particular to a model quantization method, apparatus, device, and storage medium. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on large amounts of text data. They possess strong versatility, adaptability, and reasoning capabilities in tasks such as natural language processing, multimodal understanding, and intelligent interaction. To facilitate the migration of LLMs to edge and end-device devices, they typically require quantization to reduce storage requirements and accelerate inference.

[0003] In related technologies, a small number of calibration sample datasets need to be introduced to allow the original floating-point model to propagate forward in order to collect statistical data on weights and activation values ​​of each layer in real time, including maximum and minimum values, histogram distribution, and key outlier information. The model is then quantized based on these statistical data. However, the domain distribution of calibration sample datasets is relatively simple and there is a problem of sample homogeneity, which makes it difficult to cover the complex and diverse input manifolds in real applications, resulting in performance degradation of the quantized model. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this disclosure provide a model quantization method, apparatus, device, and storage medium.

[0005] According to one aspect of the present disclosure, a model quantization method is provided, comprising:

[0006] The acquired historical lexical units are input into the large language model to be quantized, resulting in a set of candidate lexical units and the probability distribution of each candidate lexical unit.

[0007] Based on the feature space of each linear layer in the large language model to be quantized and the depth perception weight of each linear layer, the first contribution value corresponding to each candidate word is determined. The first contribution value is the contribution value of each candidate word to the latent manifold of the large language model to be quantized.

[0008] Based on the first contribution value, the target word is determined from the set of candidate word words;

[0009] The process of inputting the acquired historical lexical units into the large language model to be quantized is executed iteratively until the iteration stopping condition is met, and calibration samples are generated.

[0010] Using the calibration samples, the large language model to be quantized is quantized to obtain a quantized target model, which is then deployed to a hardware device. The quantized target model can be called by the hardware device to process corresponding tasks.

[0011] In some optional implementations, the process of obtaining the historical lexical units includes:

[0012] A lexical unit is uniformly sampled from the vocabulary of the large language model to be quantized;

[0013] Use the aforementioned lexical element as the initial historical lexical element;

[0014] Based on the initial historical lexical units and the target lexical units generated in each iteration round, new historical lexical units are generated.

[0015] In some optional implementations, determining the first contribution value corresponding to each candidate lexical unit based on the feature space of each linear layer in the large language model to be quantized and the depth-sensing weights of each linear layer includes:

[0016] Based on the feature space of each linear layer in the large language model to be quantized, the second contribution value corresponding to each candidate word is determined. The second contribution value is the contribution value of each candidate word to the geometric structure of the feature space of each linear layer.

[0017] Based on the depth perception weights of each linear layer, the second contribution values ​​corresponding to each linear layer are fused to obtain the first contribution value.

[0018] In some optional implementations, determining the second contribution value corresponding to each candidate lexical unit based on the feature space of each linear layer in the large language model to be quantized includes:

[0019] Determine the candidate lexical units and the combined lexical units formed by the historical lexical units corresponding to the candidate lexical units;

[0020] Determine the feature vector of the combined lexical unit in the linear layer;

[0021] Based on the feature vectors of the combined lexical units in the linear layer, the second contribution value corresponding to the candidate lexical unit is determined.

[0022] In some optional implementations, determining the second contribution value corresponding to the candidate lexical unit based on the feature vector of the combined lexical unit in the linear layer includes:

[0023] Based on the feature vector of the combined lexical unit in the linear layer, the manifold coverage of the combined lexical unit in the feature space of the linear layer is determined;

[0024] Logarithmic operation and second-order Taylor expansion are performed on the manifold coverage to obtain results including spectral energy and variance.

[0025] Using the results, which include spectral energy and variance, the second contribution value corresponding to the candidate lexical unit is calculated.

[0026] In some optional implementations, determining the target lexical unit from the set of candidate lexical units based on the first contribution value includes:

[0027] The probability distribution of each candidate word and its corresponding first contribution value are weighted and calculated to obtain the word score corresponding to the candidate word.

[0028] The word with the highest word score among the candidate words in the set is determined as the target word.

[0029] In some optional implementations, the iteration stopping condition includes determining the target word term a preset number of times;

[0030] The generation of calibration samples includes:

[0031] The initial historical lexical units and the target lexical units generated in each iteration are combined in sequence to obtain the calibration sample.

[0032] According to another aspect of the present disclosure, a model quantization apparatus is provided, comprising:

[0033] The acquisition module is used to input the acquired historical lexical units into the large language model to be quantized, and obtain a set of candidate lexical units and the probability distribution of each candidate lexical unit;

[0034] The first determining module is used to determine the first contribution value corresponding to each candidate word based on the feature space of each linear layer in the large language model to be quantized and the depth perception weight of each linear layer. The first contribution value is the contribution value of each candidate word to the latent manifold of the large language model to be quantized.

[0035] The second determining module is used to determine the target word from the set of candidate word words based on the first contribution value;

[0036] The sample generation module is used to iteratively execute the operation of inputting the acquired historical word units into the large language model to be quantized until the iteration stopping condition is met, and generate calibration samples.

[0037] The quantization module is used to quantize the large language model to be quantized using the calibration samples to obtain a quantized target model, which is then deployed to a hardware device. The quantized target model can be called by the hardware device to process corresponding tasks.

[0038] In some alternative implementations, it also includes:

[0039] The sampling module is used to uniformly sample a word from the vocabulary of the large language model to be quantized;

[0040] The third determining module is used to use the lexical unit as the initial historical lexical unit;

[0041] The lexical generation module is used to generate new historical lexical units based on the initial historical lexical units and the target lexical units generated in each iteration round.

[0042] In some alternative implementations, the first determining module includes:

[0043] The first determining submodule is used to determine the second contribution value corresponding to each candidate word based on the feature space of each linear layer in the large language model to be quantized. The second contribution value is the contribution value of each candidate word to the geometric structure of the feature space of each linear layer.

[0044] The fusion submodule is used to fuse the second contribution values ​​corresponding to each linear layer based on the depth perception weights of each linear layer to obtain the first contribution value.

[0045] In some alternative implementations, the first determining submodule is used for:

[0046] Determine the candidate lexical units and the combined lexical units formed by the historical lexical units corresponding to the candidate lexical units;

[0047] Determine the feature vector of the combined lexical unit in the linear layer;

[0048] Based on the feature vectors of the combined lexical units in the linear layer, the second contribution value corresponding to the candidate lexical unit is determined.

[0049] In some optional implementations, the process by which the first determining submodule determines the second contribution value corresponding to the candidate lexical based on the feature vector of the combined lexical in the linear layer may include:

[0050] Based on the feature vector of the combined lexical unit in the linear layer, the manifold coverage of the combined lexical unit in the feature space of the linear layer is determined;

[0051] Logarithmic operation and second-order Taylor expansion are performed on the manifold coverage to obtain results including spectral energy and variance.

[0052] Using the results, which include spectral energy and variance, the second contribution value corresponding to the candidate lexical unit is calculated.

[0053] In some optional implementations, the second determining module includes:

[0054] The weighting operation submodule is used to perform weighting operations on the probability distribution of each candidate word and its corresponding first contribution value to obtain the word score corresponding to the candidate word.

[0055] The second determining submodule is used to determine the word with the highest word score among the candidate word words in the set of candidate word words as the target word word word.

[0056] In some optional implementations, the iteration stopping condition includes determining the target word term a preset number of times;

[0057] The sample generation module is used to combine the initial historical lexical units and the target lexical units generated in each iteration in sequence to obtain the calibration sample.

[0058] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer program instructions that, when executed, implement the above-described method.

[0059] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0060] Memory, used to store computer program products;

[0061] A processor is used to execute a computer program product stored in memory, and when the computer program product is executed, it implements the above method.

[0062] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the above-described method.

[0063] Based on the embodiments of this disclosure, for scenarios requiring quantization of a large language model, historical lexical units can be input into the large language model to be quantized to obtain a set of candidate lexical units and their probability distributions. Then, based on the feature space of each linear layer and the depth-sensing weights of each linear layer in the large language model to be quantized, a first contribution value corresponding to each candidate lexical unit is determined. This first contribution value is the contribution of each candidate lexical unit to the latent manifold of the large language model to be quantized. Based on the first contribution value, a target lexical unit is determined from the set of candidate lexical units. The process of determining the target lexical unit is iteratively executed until the iteration stopping condition is met, generating calibration samples. Finally, the large language model to be quantized is quantized using the calibration samples to obtain the quantized target model. The quantized target model is then deployed to a hardware device, where it can be called by the hardware device to perform corresponding tasks. This achieves the generation of calibration data samples from the geometric perspective of the latent manifold of the large language model to be quantized, ensuring that the generated calibration data accurately approximates the model's true input manifold in geometric distribution. This enhances the model's generalization performance when quantizing the model based on the generated calibration samples.

[0064] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0065] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0066] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0067] Figure 1 This is a flowchart illustrating an embodiment of the model quantization method disclosed herein;

[0068] Figure 2 This is a schematic diagram illustrating the process of obtaining historical lexical units using the model quantization method disclosed herein.

[0069] Figure 3 A schematic diagram illustrating the process of determining the first contribution value of each candidate lexical unit using the model quantization method disclosed herein;

[0070] Figure 4 A schematic diagram illustrating the process of determining the second contribution value of each candidate lexical unit using the model quantization method disclosed herein;

[0071] Figure 5This is a flowchart illustrating the process of determining target lexical units using the model quantization method disclosed herein.

[0072] Figure 6 This is a schematic diagram of the model indicators after model quantization using the model quantization method disclosed herein;

[0073] Figure 7 This is a schematic diagram of the structure of one embodiment of the model quantization device of this disclosure;

[0074] Figure 8 This is a schematic diagram of the structure of yet another embodiment of the model quantization device disclosed herein;

[0075] Figure 9 This is a structural diagram of an electronic device provided as an illustrative embodiment of the present disclosure. Detailed Implementation

[0076] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0077] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0078] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0079] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0080] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0081] Furthermore, in this embodiment, the term "and / or" merely describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this embodiment, the character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0082] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0083] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0084] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0085] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0086] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0087] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0088] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0089] In order to accurately describe the technical content in this disclosure, and to accurately understand the embodiments of this disclosure, the terms used in the embodiments of this disclosure are explained or defined as follows:

[0090] 1) Token: This refers to the basic unit of input into the model, typically a word, character, or sub-word. In Natural Language Processing (NLP) tasks, tokens are the fundamental building blocks of text data. Each token represents a word, punctuation mark, or other linguistic unit. Training and / or calibration samples for the model can often include multiple tokens. In a large language model, a token can be the smallest unit for segmenting and encoding the input text; it can be a word, sub-word, character, or other form of text fragment.

[0091] 2) Input Manifold: The input manifold of a model refers to a subset of the numerical vectors corresponding to the model's input data (which can be obtained from the training data) that forms a subset with specific geometric and topological properties in the feature space. For example, when using word embedding techniques to map words in a sentence into low-dimensional dense vectors, and further representing the sentence as a vector sequence or its aggregate form, the feature representations of the words in these sentences will occupy a specific region in the high-dimensional feature space. This region may be a low-dimensional surface, curve, or other complex geometric shape, reflecting the inherent semantic, syntactic, and other structural relationships of the sentence data.

[0092] 3) Manifold Shift and Domain Bias: Real-world calibration data in related technologies typically comes from a single domain (e.g., WikiText2 mostly uses encyclopedic text). Experiments show that models calibrated using single-domain data perform extremely poorly on cross-domain tasks. For example, models calibrated using GSM8K (mathematics) exhibit severe accuracy collapse on the HumanEval (code) task. The distribution of real-world calibration data is insufficient to cover the complex and variable input manifold during the inference phase.

[0093] 4) Manifold Collapse and Sample Homogenization: Existing data-free self-generation methods (such as naive beam search) tend to generate high-probability, low-entropy repetitive patterns. From a geometric point of view, the generated samples are clustered in a few high-density regions in the feature space and cannot expand the sample space of the model.

[0094] 5) Intrinsic generative priors: In the process of machine learning modeling, these are the prior knowledge or preferences that the model itself possesses to guide the data generation process. Intrinsic generative priors are implicitly learned through the model's structure, training process, and parameter learning.

[0095] 6) Vocabulary: The set of words that the model can recognize and process, which defines the range of words involved when the model processes text data, such as the word class library and vocabulary supported by Qianwen.

[0096] 7) Contribution value of a token pair to the latent manifold of the model: The degree of influence of a token on the formation, shape, and properties of the manifold structure formed by the model input data in the feature space. A token with a high contribution value often plays a more crucial role in shaping the structure and distribution characteristics of the input manifold. The semantics, frequency of occurrence, and association with other tokens of a token will affect the magnitude of the contribution value. For example, in the processing model of an emotion analysis task, emotion keywords such as "like" and "hate" have an important impact on the distribution of texts in different emotion categories on the input manifold, and their contribution values are relatively high; some common function words (such as "of", "is", "in", etc.), although they have relatively little semantic information themselves, will also have an inescapable impact on the overall shape of the input manifold due to their extremely high frequency of occurrence; if a token often co-occurs with some other keyword tokens to form a specific semantic pattern, then it will play a greater role in maintaining the local structure and semantic coherence of the input manifold, and the contribution value will also increase accordingly.

[0097] The technical solutions involved in the embodiments of the present disclosure will be introduced in detail below.

[0098] In the current field of large-scale AI models, large-scale model quantization techniques are mainly divided into two core paths: data-free quantization and data-based quantization. Data-free quantization, often referred to as zero-sample quantization, is characterized by directly compressing the model without relying on external calibration datasets. Data-free quantization methods primarily focus on the static weight parameters in the model, calculating quantization parameters by analyzing the numerical range of the weights, or directly employing strategies such as rounding to the nearest value. Because there is no real data input, forward propagation cannot be performed to obtain the distribution statistics of dynamic activation values. Therefore, this method is usually difficult to accurately quantize activation values ​​and is mostly used in scenarios where only weights are compressed. Its advantages are that no data preparation is required, the process is extremely fast, and it is completely privacy-secure, but it often comes with some loss of accuracy. In contrast, data-based quantization is currently the mainstream method for achieving high-precision deployment. It includes a complete calibration closed-loop process. This method requires the introduction of a small amount of representative calibration dataset, allowing the original floating-point model to perform forward propagation. During this process, the system collects statistical data on weights and activation values ​​at each layer in real time, including maximum and minimum values, histogram distribution, and key outlier information. Based on these real statistical feedbacks, the algorithm can use optimization techniques such as KL divergence or mean squared error to find the optimal quantization parameters (scaling factor and zeros) and calculate the quantization error (such as perplexity changes) in pseudo-quantization simulations. If the error exceeds expectations, the system can also adjust the truncation threshold or recalibrate based on the feedback. Although data quantization is relatively complex and time-consuming, it can achieve end-to-end low-bit mapping of weights and activation values, which is a key means to ensure that the quantized model's performance is closest to the accuracy of the original model. However, existing large-scale model quantization methods still mainly select calibration samples from existing public datasets during the calibration phase. These datasets usually come from public benchmarks or task-specific corpora, and their domain distribution is relatively homogeneous. For example, commonly used calibration datasets are mostly pure text knowledge corpora, which are difficult to cover the complex and diverse input distributions in real applications. Moreover, real calibration data itself often involves data privacy issues and is difficult to fully reuse during the quantization phase.

[0099] Based on the aforementioned technical problems in related technologies, this disclosure provides a model quantization method. First, the acquired historical lexical units are input into the large language model to be quantized, resulting in a set of candidate lexical units and their probability distributions. Then, based on the feature space of each linear layer and the depth-sensing weights of each linear layer in the large language model to be quantized, a first contribution value is determined for each candidate lexical unit. This first contribution value is the contribution of each candidate lexical unit to the latent manifold of the large language model to be quantized. Next, based on the first contribution value, a target lexical unit is determined from the set of candidate lexical units. This process of determining the target lexical unit is iteratively executed until the iteration stopping condition is met, generating calibration samples. Finally, the calibration samples are used... This invention quantizes a large language model to be quantized, obtaining a quantized target model. The quantized target model is then deployed to a hardware device, where it can be invoked for corresponding tasks. This enables the generation of calibration data samples from the geometric perspective of the latent manifold of the large language model to be quantized. This ensures that the generated calibration data accurately approximates the model's true input manifold in terms of geometric distribution, thereby enhancing the model's generalization performance when quantizing the model based on the generated calibration samples. This solves the technical problems of data privacy and difficulty in covering the input manifold in related technologies that use real calibration data to quantize the model.

[0100] Exemplary methods

[0101] Figure 1 This is a flowchart illustrating an embodiment of the model quantization method of this disclosure; the model quantization method can be applied to terminal devices (such as computer devices), such as... Figure 1 As shown, the model quantization method includes the following steps 101 to 105. Each step is explained below.

[0102] In step 101, the acquired historical lexical units are input into the large language model to be quantized to obtain a set of candidate lexical units and the probability distribution of each candidate lexical unit.

[0103] Among them, the large language model to be quantized refers to a deep learning model trained on a large amount of text data, enabling the model to generate natural language text or understand language text. It can process various types of text content from the input text data, including conversational AI, chatbots, marketing content, and code assistants. A multimodal model is a model trained by combining text, images, video, audio, and other multimodal information. The input and output of a multimodal model can include multiple forms, such as text, images, audio, and video.

[0104] In this embodiment of the disclosure, the calibration data may include multiple tokens. When generating calibration data, one token of calibration data may be generated at each time step in a time step manner, and finally a calibration data is obtained.

[0105] Among them, historical lexical units are combinations of at least one lexical unit obtained before the current time step. For example, if the current time step is t, then multiple lexical units obtained before t are historical lexical units.

[0106] In practice, a word is first uniformly sampled from the vocabulary as the first word in the calibration data to be generated. Then, this first word is used as the initial historical word and input into the large language model to be quantized. A target word is generated through steps 102 to 103. The combined word formed by the target word and the first word is then determined as the historical word and input into the large language model to be quantized. This generates a new target word. When the iteration stopping condition is met, the calibration sample can be generated.

[0107] In this embodiment of the disclosure, the large language model to be quantized can utilize the model's inherent generative priors to sample the potential manifold based on historical lexical units, thereby obtaining a set of candidate lexical units and the probability distribution of each candidate lexical unit.

[0108] In practice, when the large language model to be quantized receives input historical word units, it uses internal encoding mechanisms, such as hidden state propagation in recurrent neural networks and multi-head self-attention calculation in Transformer models, to convert these word units into vector representations containing contextual information. Based on the encoded context vectors, a prediction layer (usually a fully connected layer combined with a softmax function) calculates the probability distribution of each possible candidate word unit in the current historical word unit. This prediction layer scores all word units in the vocabulary based on the context vectors, and then converts these scores into probability distributions using the softmax function. For example, when the historical word unit is "I like to eat," the model might calculate the corresponding probabilities for candidate word units such as "apple," "banana," and "noodles." The higher the probability distribution value, the greater the likelihood that the word unit will appear in the current context.

[0109] During training, large language models learn intrinsic generative priors, including word collocations, grammatical rules, and sentence structure. When generating candidate lexical units, the large language model uses these learned intrinsic generative priors to determine which lexical units are more consistent with linguistic usage in the current context. For example, in the training data, "drink" is often paired with liquid words such as "water" and "milk." When the input historical lexical unit is "I want to drink," the large language model can assign higher probability distributions to lexical units such as "water" and "milk" based on this collocation pattern.

[0110] In step 102, based on the feature space of each linear layer in the large language model to be quantized and the depth perception weights of each linear layer, the first contribution value corresponding to each candidate word is determined.

[0111] The first contribution value is the contribution of each candidate lexical to the latent manifold of the large language model to be quantized. The contribution of a candidate lexical to the latent manifold of the large language model to be quantized refers to the combination of the candidate lexical and its historical lexical, which shapes, perturbs, or locally reconstructs the topology and density distribution of the latent manifold in the high-dimensional semantic space.

[0112] In this context, the feature space of a linear layer refers to the vector space to which the input data is mapped after undergoing a linear transformation by that layer. The depth perception weights of a linear layer are related to the feature dimension of that linear layer, and can be obtained by calculating the quotient of the feature dimension of the input data of the linear layer and the overall network dimension of the large language model.

[0113] In this implementation, the first contribution value of each candidate word is determined by combining the deep perception weights of the linear layer, which helps to prevent the high-dimensional layers of the model from dominating the determination process of the target word.

[0114] In some implementations, the specific process for determining the first contribution value corresponding to each candidate lexical unit can be found in [reference needed]. Figure 3 The embodiments shown are not described in detail here.

[0115] In step 103, the target lexical is determined from a set of candidate lexicals based on the first contribution value.

[0116] In some alternative implementations, the candidate word with the largest first contribution value can be determined as the target word.

[0117] In some alternative implementations, the probability distribution of each candidate word and the first contribution value can be combined to determine the word score of the candidate word, and the candidate word with the highest word score can be determined as the target word.

[0118] In step 104, the operation of inputting the acquired historical lexical units into the large language model to be quantized is performed iteratively until the iteration stopping condition is met, and calibration samples are generated.

[0119] The process of generating calibration samples involves acquiring each word in the calibration samples step-by-step. This acquisition is primarily done iteratively. For an initial historical word, steps 101-103 above determine the target word following it. By combining the initial historical word and the target word, a new historical word is formed, and steps 101-103 above are used to obtain the new target word, until all words in the calibration samples are obtained.

[0120] By repeatedly executing steps 101 to 103, that is, using the large language model to be quantized and based on the acquired historical lexical units, the individual lexical units that make up the calibration sample can be obtained.

[0121] The iteration stopping condition may include, but is not limited to, at least one of the following: the number of times the target word is determined in the iteration reaches a preset number, or the number of characters of the historical word exceeds a preset number.

[0122] In this embodiment of the disclosure, multiple calibration samples can be obtained by repeatedly executing the above steps 101 to 104.

[0123] In this embodiment of the disclosure, a calibration sample can be obtained by sequentially combining the initial historical lexical units and the target lexical units generated in each iteration round.

[0124] In step 105, the large language model to be quantized is quantized using calibration samples to obtain the quantized target model. The quantized target model is then deployed to the hardware device, where it can be called by the hardware device to process corresponding tasks.

[0125] In this embodiment of the disclosure, a large language model can be quantized based on calibration samples using quantization techniques such as Advanced PTQ (Advanced Post-Training Quantization), Activation-aware Weight Quantization (AWQ), and Generative Pretrained Quantization (GPTQ) to obtain the quantized target model.

[0126] In practice, the calibration dataset can be input into the model in batches and forward inference can be performed. During this process, the activation value distribution data of all quantization layers is collected. Based on the collected statistical information, a specific algorithm is used to calculate the scaling factor and zero point of each layer to achieve model quantization.

[0127] Through steps 101 to 105 above, for scenarios requiring quantization of large language models, the acquired historical lexical units can be input into the large language model to be quantized to obtain a set of candidate lexical units and their probability distributions. Then, based on the feature space of each linear layer and the depth-sensing weights of each linear layer in the large language model to be quantized, the first contribution value corresponding to each candidate lexical unit is determined. This first contribution value is the contribution value of each candidate lexical unit to the latent manifold of the large language model to be quantized. Based on the first contribution value, the target lexical unit is determined from the set of candidate lexical units. The process of determining the target lexical unit is iteratively executed until the iteration stopping condition is met, generating calibration samples. Finally, the large language model to be quantized is quantized using the calibration samples to obtain the quantized target model. The quantized target model can be deployed to hardware devices, and the quantized target model can be called by the hardware devices to perform corresponding tasks. This achieves the generation of calibration data samples from the geometric perspective of the latent manifold of the large language model to be quantized, ensuring that the generated calibration data accurately approximates the model's true input manifold in geometric distribution. This enhances the generalization performance of the model when quantizing it based on the generated calibration samples.

[0128] Figure 2 This is a schematic diagram illustrating the process of obtaining historical lexical units using the model quantization method disclosed herein. For example... Figure 2 As shown, the process of obtaining historical word elements includes the following steps 201 to 203. Each step is explained below.

[0129] In step 201, a word is uniformly sampled from the vocabulary of the large language model to be quantized;

[0130] In step 202, the lexical unit is used as the initial historical lexical unit;

[0131] In step 203, new historical lexical units are generated based on the initial historical lexical units and the target lexical units generated in each iteration round.

[0132] In this embodiment of the disclosure, based on the characteristics of the model itself, each large language model to be quantized has its own corresponding vocabulary. A word can be randomly selected from the existing vocabulary of the large language model to be quantized through uniform sampling. Using the technical solution of this disclosure, calibration samples are generated based on uniformly sampled sentences.

[0133] In one example, a word can be uniformly sampled from the vocabulary of the large language model to be quantized, and the sampled word can be used as the initial historical word. Then, the subsequent target word can be obtained by iteratively executing steps 101 to 103. Finally, the initial historical word and the subsequent target word are concatenated to form a calibration sample.

[0134] Based on the embodiments of this disclosure, by uniformly sampling a lexical unit from the vocabulary of a large language model as the starting lexical unit for calibration samples, the unbiasedness of the initial distribution can be ensured. The samples generated using autoregression are more closely related to the training data in terms of distribution, ensuring that the generated calibration samples cover all aspects and have strong generalization. Furthermore, the calibration samples generated by autoregression have good coherence, making full use of the model's own vocabulary, with relatively coherent internal words, resulting in a more natural and fluent generation. Moreover, the calibration samples generated using autoregression have sufficiently low perplexity on the large language model to be quantized, closely approximating the output distribution of the large language model to be quantized.

[0135] Perplexity is a common metric for evaluating the performance of language models in Natural Language Processing (NLP). Perplexity reflects the predictive ability of a language model for a given text; a lower perplexity value indicates a stronger predictive ability. Calibration samples generated using an autoregressive approach exhibit significantly lower perplexity than other calibration datasets, thus improving the model's predictive power.

[0136] Figure 3 This is a schematic diagram illustrating the process of determining the first contribution value of each candidate lexical unit in the model quantization method of this disclosure. For example... Figure 3 As shown, the process of obtaining the first contribution value includes the following steps 301 to 302. Each step is explained below.

[0137] In step 301, based on the feature space of each linear layer in the large language model to be quantized, the second contribution value corresponding to each candidate word is determined;

[0138] In step 302, based on the depth perception weights of each linear layer, the second contribution values ​​corresponding to each linear layer are fused to obtain the first contribution value.

[0139] The second contribution value is the contribution of each candidate lexical unit to the geometric structure of the feature space of each linear layer. The contribution of a candidate lexical unit to the feature space of each linear layer refers to the combination of the candidate lexical unit and its historical lexical units, which shapes, perturbs, or locally reconstructs the topology and density distribution of the feature space in the feature space of the input data of the corresponding linear layer.

[0140] In this context, the feature space of a linear layer refers to the vector space to which the input data is mapped after undergoing a linear transformation by that layer.

[0141] Among them, the depth perception weight of the linear layer is related to the feature dimension of the linear layer, which can be obtained by calculating the quotient of the feature dimension of the input data of the linear layer and the overall network dimension of the large language model.

[0142] Based on the embodiments of this disclosure, by first determining the second contribution value of each candidate word and then combining it with the deep perception weight of the linear layer to determine the first contribution value of each candidate word, it helps to prevent the high-dimensional layer of the model from dominating the determination process of the target word and improves the accuracy of the determined target word.

[0143] Figure 4 This is a schematic diagram illustrating the process of determining the second contribution value of each candidate lexical unit in the model quantization method of this disclosure. For example... Figure 4 As shown, the process of obtaining the second contribution value includes the following steps 401 to 403. Each step is explained below.

[0144] In step 401, candidate lexical units and the combined lexical units formed by the historical lexical units corresponding to the candidate lexical units are determined;

[0145] In step 402, the feature vector of the combined word unit in the linear layer is determined;

[0146] In step 403, the second contribution value corresponding to the candidate word is determined based on the feature vector of the combined word in the linear layer.

[0147] One approach is to first connect candidate lexical units and their corresponding historical lexical units in sequence to obtain combined lexical units or short phrases.

[0148] When determining the second contribution value of candidate lexical units, the manifold coverage of the combined lexical units in the feature space of the linear layer can be determined based on the feature vector of the combined lexical units in the linear layer. Logarithmic operation and second-order Taylor expansion are performed on the manifold coverage to obtain the results including spectral energy and variance. The second contribution value corresponding to the candidate lexical unit is calculated using the results including spectral energy and variance.

[0149] First, the manifold coverage of each candidate word to the geometric structure of the feature space of the corresponding linear layer in the large language model to be quantized can be calculated using Equation (1).

[0150] Equation (1)

[0151] In equation (1), X is the feature vector of the combined word unit in the linear layer, which belongs to The feature vector, where N is the number of calibration samples to be generated. It represents the feature dimension of the input data in the linear layer; det stands for determinant operation, used to measure the "volume scaling factor" of the linear transformation corresponding to the matrix. This represents manifold coverage.

[0152] By formula (2) Use a more convenient descriptive format.

[0153] Equation (2)

[0154] Logarithmic calculation of manifold coverage is performed using equation (3).

[0155] Equation (3)

[0156] In equation (3), λ represents the eigenvalue of G. It is the feature dimension of the input data represented in the linear layer.

[0157] By using equation (4) to modify equation (3) Performing a second-order Taylor expansion yields results including spectral energy and variance.

[0158] Equation (4)

[0159] In equation (4), λ represents the eigenvalue of G. It is a common parameter in Taylor's formula.

[0160] Substituting equation (4) into equation (3) and summing the results, we can obtain equation (5).

[0161] Equation (5)

[0162] Furthermore, the trace of a matrix can be used. Further transformation of equation (5) yields equation (6).

[0163] Equation (6)

[0164] In equation (6), This indicates the spectral energy that needs to be amplified. This indicates the variance that needs to be reduced. Reducing the variance helps maintain the isotropy of the spectrum.

[0165] By utilizing the trace of the G matrix as a metric for the upper bound of the energy and volume of the feature space, and treating it as a reward signal in the generation process, the spectral energy can be amplified; by utilizing the Frobenius norm and trace of the Gram matrix... The ratio of is used as a proxy index for the flatness (isotropy) of the spectral distribution and as a penalty term in the generation process, which can reduce variance and ensure the maintenance of the isotropy of the spectrum.

[0166] In some alternative implementations, equivalent substitutions can be used. Regarding the variance in equation (6) Further transformations are made, such as equations (7) and (8).

[0167] Equation (7)

[0168] Equation (8)

[0169] Ignore the constant term in equation (8) It can Confirmed as a peak penalty item .

[0170] By transforming and equivalently substituting the above formulas, the second contribution value corresponding to each linear layer can be obtained. .

[0171] In summary, based on the embodiments of this disclosure, when determining the second contribution value of each candidate lexical unit, a spectral energy maximization strategy is used, employing the trace as a measure of the volume upper limit, to force the generated activation vector to expand radially in the feature space. This allows the generated calibration data to fully "expand" the feature space, covering a wider dynamic range, thereby significantly improving the robustness of quantization parameter estimation. Furthermore, by introducing a spectral isotropic regularization strategy, the peak penalty term is minimized, forcing the feature value distribution to be uniform, preventing the manifold from over-extending or collapsing in a single dimension. Moreover, through second-order Taylor expansion, the complex volume optimization objective is creatively decoupled into the operation of trace and Frobenius norm, reducing the computational complexity to O(d^2), which helps to complete the generation of calibration data for models with large parameter scales in a shorter time, demonstrating extremely high engineering practicality.

[0172] Extensive experimental data have demonstrated that, under different random seed (initial historical lexical) initializations, the calibration data generated using the disclosed technical solution exhibits extremely low standard deviation in quantization accuracy, ensuring high consistency of results during industrial deployment.

[0173] Figure 5 This is a flowchart illustrating the process of determining target lexical units using the model quantization method disclosed herein. For example... Figure 5 As shown, the process of determining the target lexical may include steps 501 to 502. Each step is explained below.

[0174] In step 501, the probability distribution of each candidate word and its corresponding first contribution value are weighted and calculated to obtain the word score corresponding to the candidate word.

[0175] In step 502, the word with the highest word score among a group of candidate words is determined as the target word.

[0176] In passing Figure 4After determining the second contribution value in the illustrated embodiment, the second contribution values ​​corresponding to each linear layer can be summed to obtain the first contribution value. Then, combined with the probability distribution of each candidate word, the word score of each candidate word is determined. Thus, the final calibration sample generated from the target word obtained based on the word score can ensure both fluency and compliance with geometric indicators (ensuring the robustness of model quantization).

[0177] Specifically, the word score can be determined using the following formula:

[0178] Equation (9)

[0179] In equation (9), L is the current linear layer number, and L is the total number of layers. It is the lexical score of the lexical obtained at time step t. It is the probability distribution of candidate word elements. Let D represent the feature dimension of each linear layer, and let D represent the total dimension of the large language model to be quantized. This indicates that normalization is being performed. It is a constant, which can be set through experiments. It is 1.2.

[0180] After determining the word score of each candidate word using equation (9), the word with the highest word score can be identified as the target word.

[0181] Based on the embodiments of this disclosure, by combining language model probability, normalized spectral energy, and normalized isotropic penalty, and weighting the lexical scores according to the hierarchical dimension d, the effectiveness of the determined target lexicals is improved, which helps to obtain high-quality calibration data. By utilizing the intrinsic generative prior of the large language model to be quantized, a high-quality calibration set can be generated without accessing any external datasets. This solves the problem of obtaining high-quality calibration data in sensitive fields such as finance and healthcare due to privacy compliance restrictions, and greatly reduces the threshold for model quantization deployment.

[0182] Figure 6The diagram illustrates the metrics of a large language model after quantization of calibration data generated using the technical solution of this disclosure. The IVSE (iso-volume self-expansion) calibration samples are generated using the technical solution of this disclosure. The wiki, mmlu (Massive Multitask Language Understanding) benchmark, gsm8k (Grade School Math 8K, an English math word problem dataset), and humaneval (a dataset specifically used to evaluate the code generation capabilities of large language models) are real-world calibration datasets from various domains. The PPL (Perplexity), humaneval, gsm8k_cot, mmlu, and mbpp items in the top row of the table indicate the model's performance metrics.

[0183] pass Figure 6 As can be seen, the performance indicators of the large language model after quantization of the calibration data generated using the disclosed technical solution are most similar to those of the floating-point model before quantization, and both models have good performance across various performance indicators, indicating that the predictions after quantization are more accurate.

[0184] The model quantization method provided in this disclosure is applicable to quantizing large language models deployed on electronic devices, especially for quantizing large language models on edge devices. Edge devices include various computing devices deployed on edge nodes, such as smartphones, smart home devices, and cameras. Edge devices are located closer to the data source and user, and have higher real-time requirements; quantization enables the model to meet these requirements.

[0185] The embodiments disclosed herein can improve inference performance on edge testing devices while enhancing the generalization ability of large language models.

[0186] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0187] Corresponding to the aforementioned embodiments of the model quantization method, this disclosure also provides embodiments of the model quantization apparatus.

[0188] Exemplary device

[0189] Figure 7This is a schematic diagram of one embodiment of the model quantization device disclosed herein, which is applied to electronic devices (such as computer equipment, servers, etc.). Figure 7 As shown, the model quantization device may include:

[0190] The acquisition module 71 is used to input the acquired historical word units into the large language model to be quantized, and obtain a set of candidate word units and the probability distribution of each candidate word unit;

[0191] The first determining module 72 is used to determine the first contribution value corresponding to each candidate word based on the feature space of each linear layer in the large language model to be quantized and the depth perception weight of each linear layer. The first contribution value is the contribution value of each candidate word to the potential manifold of the large language model to be quantized.

[0192] The second determining module 73 is used to determine the target word from a set of candidate word words based on the first contribution value;

[0193] The sample generation module 74 is used to iteratively execute the operation of inputting the acquired historical word units into the large language model to be quantized until the iteration stopping condition is met, and to generate calibration samples.

[0194] The quantization module 75 is used to quantize the large language model to be quantized using calibration samples to obtain the quantized target model, which can then be deployed to the hardware device for processing corresponding tasks.

[0195] Figure 8 This is a schematic diagram of another embodiment of the model quantization device disclosed herein. Figure 8 As shown, in Figure 7 Based on the illustrated embodiments, in some optional implementations, the model quantization apparatus may further include:

[0196] The sampling module 76 is used to uniformly sample a word from the vocabulary of the large language model to be quantized;

[0197] The third determining module 77 is used to use lexical units as initial historical lexical units;

[0198] The lexical generation module 78 is used to generate new historical lexical units based on the initial historical lexical units and the target lexical units generated in each iteration round.

[0199] In some alternative implementations, the first determining module 72 includes:

[0200] The first determining submodule 721 is used to determine the second contribution value corresponding to each candidate word based on the feature space of each linear layer in the large language model to be quantized. The second contribution value is the contribution value of each candidate word to the geometric structure of the feature space of each linear layer.

[0201] The fusion submodule 722 is used to fuse the second contribution values ​​corresponding to each linear layer based on the depth perception weights of each linear layer to obtain the first contribution value.

[0202] In some optional implementations, the first determining submodule 721 is used to: determine candidate lexical units and the combined lexical units formed by the historical lexical units corresponding to the candidate lexical units; determine the feature vector of the combined lexical units in the linear layer; and determine the second contribution value corresponding to the candidate lexical unit based on the feature vector of the combined lexical units in the linear layer.

[0203] In some optional implementations, the process by which the first determining submodule 721 determines the second contribution value corresponding to the candidate word based on the feature vector of the combined word in the linear layer may include: determining the manifold coverage of the combined word in the feature space of the linear layer based on the feature vector of the combined word in the linear layer; performing logarithmic operation and second-order Taylor expansion on the manifold coverage to obtain a result including spectral energy and variance; and using the result including spectral energy and variance to calculate the second contribution value corresponding to the candidate word.

[0204] In some alternative implementations, the second determining module 73 includes:

[0205] The weighting operation submodule 731 is used to perform weighting operations on the probability distribution of each candidate word and its corresponding first contribution value to obtain the word score corresponding to the candidate word.

[0206] The second determination submodule 732 is used to determine the word with the highest word score among a set of candidate word words as the target word word.

[0207] In some optional implementations, the iteration stopping condition includes the number of times the target word has been determined in the iteration reaches a preset number;

[0208] The sample generation module 74 is used to combine the initial historical lexical units and the target lexical units generated in each iteration in sequence to obtain calibration samples.

[0209] The modules and units in this disclosed device can be further divided into finer-grained units according to actual needs, and the specific configuration can be set according to actual needs.

[0210] The apparatus of this disclosure embodiment can be used to implement the methods of the above embodiments of this disclosure. The two correspond to each other in specific implementation, and the specific implementation of related parts can be referred to each other, which will not be repeated here.

[0211] Exemplary electronic devices, computer program products, and computer-readable storage media

[0212] This disclosure also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the model quantization method of any of the above embodiments of this disclosure.

[0213] Below, for reference Figure 9 This describes an electronic device according to embodiments of the present disclosure, wherein apparatus for implementing methods according to embodiments of the present disclosure may be integrated. Figure 9 This is a structural diagram of an electronic device provided in an illustrative embodiment of the present disclosure, such as... Figure 9 As shown, the electronic device includes one or more processors 91, one or more memory 92s of computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 92 is executed, the aforementioned model quantization method can be implemented.

[0214] Specifically, in practical applications, the electronic device may also include components such as an input device 93 and an output device 94, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will understand that... Figure 9 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or certain components, or different component arrangements. Wherein:

[0215] The processor 91 may be a central processing unit (CPU) or other processing unit with model quantization capability and / or instruction execution capability. It performs various functions and processes data by running or executing software programs and / or modules stored in memory 92 and calling data stored in memory 92, thereby performing overall monitoring of the electronic device.

[0216] Memory 92 can store one or more computer program products. The memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and processor 91 can run the computer program products to implement the model quantization methods of the various embodiments of this disclosure above and / or other desired functions.

[0217] The input device 93 can be used to receive input numeric or character information. The input device 93 may include a keyboard, mouse, joystick, etc., related to user settings and function control.

[0218] The output device 94 can output various information to the outside, including determined distance information, direction information, etc. The output device 94 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0219] Electronic devices may also include a power supply for powering various components, which can be logically connected to the processor 91 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply may also include one or more DC or AC power sources, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and any other components.

[0220] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0221] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model quantization methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0222] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0223] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the model quantization methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0224] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0225] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in these embodiments are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0226] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0227] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0228] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustrative purposes only, and the steps of the method of this disclosure are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the method according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the method according to this disclosure.

[0229] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A model quantization method, comprising: The acquired historical lexical units are input into the large language model to be quantized, resulting in a set of candidate lexical units and the probability distribution of each candidate lexical unit. Based on the feature space of each linear layer in the large language model to be quantized and the depth perception weight of each linear layer, the first contribution value corresponding to each candidate word is determined. The first contribution value is the contribution value of each candidate word to the latent manifold of the large language model to be quantized. Based on the first contribution value, the target word is determined from the set of candidate word words; The process of inputting the acquired historical lexical units into the large language model to be quantized is executed iteratively until the iteration stopping condition is met, and calibration samples are generated. Using the calibration samples, the large language model to be quantized is quantized to obtain a quantized target model, which is then deployed to a hardware device. The quantized target model can be called by the hardware device to process corresponding tasks.

2. The method according to claim 1, characterized in that, The process of obtaining the historical lexical units includes: A lexical unit is uniformly sampled from the vocabulary of the large language model to be quantized; Use the aforementioned lexical units as initial historical lexical units; Based on the initial historical lexical units and the target lexical units generated in each iteration round, new historical lexical units are generated.

3. The method according to any one of claims 1-2, characterized in that, The method for determining the first contribution value corresponding to each candidate lexical unit based on the feature space of each linear layer in the large language model to be quantized and the depth perception weights of each linear layer includes: Based on the feature space of each linear layer in the large language model to be quantized, the second contribution value corresponding to each candidate word is determined. The second contribution value is the contribution value of each candidate word to the geometric structure of the feature space of each linear layer. Based on the depth perception weights of each linear layer, the second contribution values ​​corresponding to each linear layer are fused to obtain the first contribution value.

4. The method according to claim 3, characterized in that, The determination of the second contribution value corresponding to each candidate lexical unit based on the feature space of each linear layer in the large language model to be quantized includes: Determine the candidate lexical units and the combined lexical units formed by the historical lexical units corresponding to the candidate lexical units; Determine the feature vector of the combined lexical unit in the linear layer; Based on the feature vectors of the combined lexical units in the linear layer, the second contribution value corresponding to the candidate lexical unit is determined.

5. The method according to claim 4, characterized in that, The step of determining the second contribution value corresponding to the candidate word based on the feature vector of the combined word in the linear layer includes: Based on the feature vector of the combined lexical unit in the linear layer, the manifold coverage of the combined lexical unit in the feature space of the linear layer is determined; Logarithmic operation and second-order Taylor expansion are performed on the manifold coverage to obtain results including spectral energy and variance. Using the results, which include spectral energy and variance, the second contribution value corresponding to the candidate lexical unit is calculated.

6. The method according to any one of claims 1-5, characterized in that, The step of determining the target word from the set of candidate word words based on the first contribution value includes: The probability distribution of each candidate word and its corresponding first contribution value are weighted and calculated to obtain the word score corresponding to the candidate word. The word with the highest word score among the candidate words in the set is determined as the target word.

7. The method according to any one of claims 1-6, characterized in that, The iteration stopping condition includes the number of times the target word element is determined in the iteration reaches a preset number; The generation of calibration samples includes: The initial historical lexical units and the target lexical units generated in each iteration are combined in sequence to obtain the calibration sample.

8. A model quantization device, comprising: The acquisition module is used to input the acquired historical lexical units into the large language model to be quantized, and obtain a set of candidate lexical units and the probability distribution of each candidate lexical unit; The first determining module is used to determine the first contribution value corresponding to each candidate word based on the feature space of each linear layer in the large language model to be quantized and the depth perception weight of each linear layer. The first contribution value is the contribution value of each candidate word to the latent manifold of the large language model to be quantized. The second determining module is used to determine the target word from the set of candidate word words based on the first contribution value; The sample generation module is used to iteratively execute the operation of inputting the acquired historical word units into the large language model to be quantized until the iteration stopping condition is met, and generate calibration samples. The quantization module is used to quantize the large language model to be quantized using the calibration samples to obtain a quantized target model, which is then deployed to a hardware device. The quantized target model can be called by the hardware device to process corresponding tasks.

9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

11. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

Images