Quantization method and system of hybrid expert big language model, medium and equipment

Abstract

The invention provides a quantification method and system of a hybrid expert large language model, a medium and equipment, and the method comprises the steps: carrying out the forward propagation of a to-be-quantified hybrid expert large language model, and determining an activation value of each layer; according to the activation value of each layer, Fisher information of each expert weight is determined; according to the Fisher information of the weight of each expert, performing linear weighted fusion on the weight of each expert, and determining a shared basic weight; taking a difference value between the weight of each expert and the shared basic weight as a residual weight of each expert; and carrying out weight binarization processing on the shared basic weight by adopting a preset alternative refining binarization strategy, carrying out weight binarization processing on the residual weight of each expert by adopting a mode of combining a symbol matrix and a scaling factor, and determining a quantized hybrid expert large language model. By means of the method and device, effective separation and compression of expert generality and personality information are achieved, the compression precision and the compression rate are effectively balanced, and the calculation cost and the deployment cost are reduced.

Description

Technical Field

[0001] This application relates to the field of model quantization technology, specifically to a quantization method, system, medium, and device for a hybrid expert large language model. Background Technology

[0002] In recent years, with the introduction and continuous evolution of the Transformer architecture, Large Language Models (LLMs) have made groundbreaking progress in the field of Natural Language Processing (NLP). From the BERT series to representative models such as LLaMA, their scale has rapidly expanded from hundreds of millions to hundreds of billions, raising the performance ceiling for many tasks such as text generation, question answering, translation, and code generation. This wave of model-driven paradigm shift benefits from the extremely strong fit of ultra-large-scale model parameters to language knowledge and contextual understanding. However, the leap in performance has also brought significant computational and resource costs.

[0003] Current mainstream Large Language Models (LLMs) place extremely high demands on GPU memory and computing power during training and inference. For example, the LLaMA3-70B requires over 150GB of GPU memory during inference, far exceeding the available capacity of most devices. As the number of model layers, dimensions, and context length continue to expand, LLMs are increasingly revealing significant deployment bottlenecks in tasks such as multi-turn interactions, long text generation, and low-latency responses. Especially in mobile, edge device, and low-power environments, the storage size and runtime overhead of the models have become core obstacles restricting their large-scale deployment. Therefore, how to reduce the storage size and inference cost of LLMs while preserving their expressive power as much as possible has become one of the important research issues.

[0004] To alleviate these challenges, researchers have proposed various model compression techniques, including weight quantization, low-rank decomposition, structural pruning, and knowledge distillation. Among these, binarization, as an extreme quantization method, can compress model parameters to 1 bit, significantly reducing storage and computational overhead, theoretically achieving a compression ratio of 32×. Compared to large model quantization methods that require retraining, post-training quantization (PTQ) methods have become mainstream due to their efficiency, lightweight nature, and lack of backpropagation requirements. Recent methods such as BiLLM and ARB-LLM have significantly narrowed the performance gap between binarized models and full-precision models through strategies such as mixed precision, alternating optimization, and distribution alignment.

[0005] For example, the patent "CN118036661B; Mixed-precision quantization method, device, electronic device and medium for large language models" separates outliers and normal data, allocates appropriate bit width according to weight sensitivity, and improves the algorithm accuracy and hardware resource utilization of the model through mixed-precision quantization, thereby narrowing the performance gap between the binarized model and the full-precision model.

[0006] Meanwhile, hybrid expert models (MoEs) are gradually becoming one of the key architectural designs for improving model capacity and computational efficiency. MoEs introduce multiple expert subnetworks, activating only a small number of experts in each forward computation to participate in reasoning, thus achieving modeling capabilities for extremely large parameter spaces with relatively low computational cost. This sparse activation mechanism enables MoEs to maintain near-linear computational overhead while the number of parameters increases. Typical MoEs, such as Switch Transformer, Mixtral, DeepSeek-MoE, and Phi-MoE, have achieved significant breakthroughs in tasks such as language modeling and dialogue generation.

[0007] However, despite the sparsity of the Hybrid Expert Model (MoE) architecture at the inference computation level, its overall parameter size remains enormous. Each expert subnetwork is essentially a complete feedforward module, and all expert parameters must be pre-loaded or reside in GPU memory, resulting in high storage and loading costs. This parameter redundancy severely limits the widespread application of Hybrid Expert Models (MoE) in lightweight deployments and resource-constrained environments.

[0008] To address this, existing work has attempted to compress hybrid expert models (MoE) using two main methods: expert pruning and expert merging. Pruning methods, such as MoE-I2 and NAEE, reduce the number of parameters by removing some experts or their internal weights, but this may compromise the expertise of the experts, leading to a significant performance degradation. Expert merging methods, such as EEP and MC-some, attempt to merge multiple experts into a shared subnetwork, but they often rely on the assumption of highly overlapping expert functions, making them difficult to generalize in real-world scenarios and potentially compromising model diversity and generalization ability.

[0009] Therefore, using unified quantization for hybrid expert models leads to a severe decline in the performance of highly sensitive experts, while full-precision storage cannot meet deployment requirements. Furthermore, there are a large number of redundant representations among experts, and direct pruning can easily destroy the model's functionality. Experimental results show that an average overall bit width of 1.3 bits per parameter is achieved on Mixtral-8×7B, which improves the downstream task accuracy by 8.7% compared to full 1-bit quantization, while reducing the feature overlap among experts by 32%.

[0010] The existing hybrid expert model (MoE) compression methods struggle to balance efficiency and accuracy, making it a critical issue that urgently needs to be addressed in the field of model quantization. Summary of the Invention

[0011] In view of the deficiencies in the prior art, the purpose of this application is to provide a quantification method, system, medium and device for hybrid expert large language models.

[0012] A first aspect of this application provides a quantification method for hybrid expert large language models, comprising: A preset number of input samples are input into the hybrid expert large language model to be quantized, and forward propagation is performed to determine the activation value of each layer of the hybrid expert large language model to be quantized. Based on the activation value of each layer, determine the Fisher information for each expert weight; Based on the Fisher information of each expert weight, the expert weights are linearly weighted and fused to determine the shared basic weights. The difference between each expert's weight and the shared basic weight is used as the residual weight of each expert. The shared basic weights are binarized using a preset alternating refinement binarization strategy, and the residual weights of each expert are binarized using a combination of symbol matrix and scaling factor to determine the quantized hybrid expert large language model.

[0013] Optionally, the method further includes: Based on the Fisher information of each expert's weight, an importance analysis is performed on the residual weight of each expert to determine the importance of the residual weight of each expert. Based on the importance of the residual weights of each expert, the binarization order of the residual weights of each expert is assigned, thereby determining the binarization order of the residual weights of each expert.

[0014] Optionally, determining the Fisher information for each expert weight based on the activation value of each layer includes: Based on the input activation of each layer and the output activation of each expert, a gradient approximation of the weights of each expert is constructed. The element-wise square of the gradient of each expert weight is used as a preliminary estimate of the weight sensitivity of each expert weight. The overall magnitude of the input activation at each layer is used to normalize the preliminary estimate of the weight sensitivity of each expert weight, thereby determining the weight sensitivity estimation result of each expert weight. The Fisher information for each expert weight is determined by averaging the weight sensitivity estimation results for each expert weight.

[0015] Optionally, the step of performing weight binarization processing on the shared basic weights using a preset alternating refinement binarization strategy includes: The shared basic weights are initially binarized to generate a first symbol matrix, an initial first row scaling factor, and an initial first column scaling factor. Based on the first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor, determine the initial binary approximate representation of the shared basic weights; Based on the reconstruction error between the shared basic weights and the initial binary approximation representation, a first residual term is constructed; In the next round of alternating refining, the first residual term from the previous round of alternating refining is used as the optimization target. A greedy parameter update strategy is adopted to select the update direction with the largest decrease in the first residual term, and the first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor are updated to determine the new first symbol matrix, the new first row scaling factor, and the new first column scaling factor. In the next round of alternating refining, a new binarized approximate representation of the shared basic weights is determined based on the new first symbol matrix, the new first row scaling factor, and the new first column scaling factor. In the next round of alternating refining, a new first residual term is constructed based on the reconstruction error between the shared basic weights and the new binary approximation representation; The process involves multiple rounds of alternating refinement until the difference between the new binarized approximate representations of the shared basic weights in two adjacent rounds of alternating refinement is less than a preset first critical value or reaches a preset first iteration round, at which point the binarization result of the shared basic weights is determined.

[0016] Optionally, the step of performing weight binarization on the residual weights of each expert using a combination of a sign matrix and a scaling factor includes: The residual weights of each expert are initially binarized to generate a second symbol matrix, an initial second row scaling factor, and an initial second column scaling factor. Based on the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor, determine the initial binarized approximate representation of the residual weights of each expert; A second residual term is constructed based on the reconstruction error between the residual weight of each expert and the initial binary approximation representation of the residual weight of each expert. In the next round of alternating refining, the second residual term from the previous round of alternating refining is used as the optimization target. A greedy parameter update strategy is adopted to select the update direction with the largest decrease in the second residual term, and the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor are updated to determine the new second symbol matrix, the new second row scaling factor, and the new second column scaling factor. In the next round of alternating refining, a new binarized approximate representation of the residual weights of each expert is determined based on the new second symbol matrix, the new second row scaling factor, and the new second column scaling factor. In the next round of alternating refining, a new second residual term is constructed based on the reconstruction error between the residual weight of each expert and the new binarized approximate representation of the residual weight of each expert. The process involves multiple rounds of alternating refinement until the difference between the new binarized approximate representations of the residual weights of each expert in two adjacent rounds of alternating refinement is less than a preset second critical value or a preset second iteration is reached. The binarization result of the residual weights of each expert is then determined.

[0017] A second aspect of this application provides a quantization system for a hybrid expert large language model, comprising: The activation value acquisition module is used to input a preset number of input samples into the hybrid expert large language model to be quantized, perform forward propagation, and determine the activation value of each layer of the hybrid expert large language model to be quantized. The Fisher information extraction module is used to determine the Fisher information for each expert weight based on the activation value of each layer. The shared basic weight determination module is used to perform linear weighted fusion on each expert weight based on the Fisher information of each expert weight to determine the shared basic weight. The residual weight determination module is used to take the difference between each expert weight and the shared basic weight as the residual weight of each expert. The weight binarization module is used to perform weight binarization on the shared basic weights using a preset alternating refinement binarization strategy, and to perform weight binarization on the residual weights of each expert using a combination of symbol matrix and scaling factor, so as to determine the quantized hybrid expert large language model.

[0018] A third aspect of this application provides a text generation method based on a hybrid expert large language model, comprising: Acquire natural language prompts, context sequences, instruction text, historical dialogues, and a text generation model to be quantized; The text generation model to be quantized is quantized using the quantization method of the hybrid expert large language model provided in any of the first aspects of this application, and the quantized text generation model is determined. The natural language prompts, the context sequence, the instruction text, and the historical dialogue are input into the quantized text generation model to determine the text generation result.

[0019] A fourth aspect of this application provides a knowledge extraction method based on a hybrid expert large language model, comprising: Obtain the text to be analyzed, factual questions, entity relationship queries, retrieval prompts, contextual text, and knowledge extraction models; The knowledge extraction model is quantified using the quantification method of the hybrid expert big language model provided in any of the first aspects of this application to determine the quantified knowledge extraction model. The text to be analyzed, the factual question, the entity relationship query, the retrieval prompt, and the context text are input into the quantified knowledge extraction model to determine the extracted knowledge.

[0020] A fifth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0021] A sixth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods provided in the first aspect of this application.

[0022] The quantization method of the hybrid expert large language model in this application guides the linear weighted fusion of each expert weight based on Fisher information to extract shared basic weights, thereby improving the initial quality of quantization compression. The difference between each expert weight and the shared basic weight is used as the residual weight of each expert, explicitly modeling the individual differences between experts, fully preserving the individual expressive ability of experts, and preventing quantization performance degradation. A preset alternating refinement binarization strategy is used to perform weight binarization processing on the shared basic weights to optimize the binarization quality of the shared basic weights and improve the performance preservation capability of the compressed model. The residual weights of each expert are binarized layer by layer using a combination of sign matrix and scaling factor to further improve the compression ratio. Through an adaptive residual fitting binarization algorithm that includes a preset alternating refinement binarization strategy and a combination of sign matrix and scaling factor, the accuracy and compression ratio are effectively balanced, improving the deployment efficiency of the hybrid expert large language model on edge devices. Moreover, the entire process requires no retraining, achieving plug-and-play functionality, reducing deployment costs, and improving the feasibility of industrial applications.

[0023] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0024] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a quantization method for a hybrid expert large language model according to an exemplary embodiment.

[0025] Figure 2 This is a schematic diagram illustrating the structure of a quantization system for a hybrid expert large language model according to an exemplary embodiment. Detailed Implementation

[0026] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0027] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0028] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0029] Existing methods for compressing hybrid expert models include expert pruning and expert merging. Pruning methods, such as MoE-I2 and NAEE, reduce the number of parameters by removing some experts or their internal weights, but this compromises the expertise of the experts and reduces the performance of the compressed model. Expert merging methods, such as EEP and MC-some, fuse multiple experts into a shared subnetwork, which is difficult to generalize in real-world scenarios. To address these issues, this application provides a quantization method for hybrid expert large language models to resolve the aforementioned problems.

[0030] Figure 1 This is a flowchart illustrating a quantization method for a hybrid expert large language model according to an exemplary embodiment.

[0031] Reference Figure 1 As shown in one embodiment of this application, a quantization method for a hybrid expert large language model is provided, including steps S11 to S15.

[0032] S11, input a preset number of input samples into the hybrid expert large language model to be quantized, perform forward propagation, and determine the activation value of each layer of the hybrid expert large language model to be quantized.

[0033] Specifically, a preset number of input samples can be extracted from publicly available corpora, such as selecting several samples from general natural language corpora like WikiText, C4, and Pile that can cover a variety of sentence structures, vocabulary, and themes as input samples.

[0034] The activation value of each layer includes the input activation value X of each layer and the output activation value Y of each expert.

[0035] S12, determine the Fisher information for each expert weight based on the activation value of each layer.

[0036] Specifically, Fisher information is a measure of how sensitive the model output is to parameter perturbations, used to represent the degree of influence of a certain expert weight on the overall output of the hybrid expert large language model.

[0037] The larger the value of the Fish information, the more sensitive and important the expert weight is to the model output.

[0038] S13. Based on the Fisher information of each expert weight, perform linear weighted fusion on each expert weight to determine the shared basic weight.

[0039] S14, the difference between each expert's weight and the shared basic weight is used as the residual weight of each expert.

[0040] S15, a preset alternating refinement binarization strategy is used to perform weight binarization on the shared basic weights, and a combination of symbol matrix and scaling factor is used to perform weight binarization on the residual weights of each expert, thus determining the quantized hybrid expert large language model.

[0041] The pre-defined alternating refinement binarization strategy is to perform multiple rounds of iterative binarization approximation without the need for further training.

[0042] Specifically, in each iteration, the sign matrix is ​​first updated based on the sign information of the current weights (or the current residuals). Then, the row scaling factor and column scaling factor are updated based on the statistical information of the row and column directions, respectively, to obtain a new binarized approximate representation. Subsequently, a new residual term is constructed based on the reconstruction error between the original weights and this binarized approximate representation, and the process proceeds to the next iteration. This process is repeated until the change in the binarized approximate representation obtained from two adjacent iterations is less than a preset critical value, or the preset upper limit of the number of iterations is reached, thus outputting the binarized result of the original weights. To improve convergence stability, a greedy parameter update strategy can be adopted in each iteration, prioritizing the update direction that results in a greater reduction in reconstruction error.

[0043] The combination of the sign matrix and scaling factor is as follows: for any weight matrix to be quantized, which has an shape of m×n, it is split into two parts: the "sign structure" and the "scaling factor". The sign structure is represented by the sign matrix B, which also has an shape of m×n. Each element of the sign matrix B takes only two sign values ​​(positive or negative), so the sign matrix B can be stored using 1-bit elements to record the sign information of the corresponding weight elements. The scaling factor is used to adjust the amplitude of each sign element in the sign matrix B, thereby obtaining an approximate representation of the original weight matrix.

[0044] Specifically, the scaling factors include at least two types: row scaling factors and column scaling factors. Row scaling factors are vectors of length m, used to scale the overall magnitude of each row of the symbol matrix B. Column scaling factors are vectors of length n, used to scale the overall magnitude of each column of the symbol matrix B. The row and column scaling factors, combined with the symbol matrix B in a fixed order, yield an approximate matrix of the same shape (m×n) as the original weight matrix. This approximate matrix is ​​identical to the symbol matrix B in terms of sign, and its magnitude is jointly determined by the row and column scaling factors. For storage and deployment, the symbol matrix B uses 1-bit encoding to achieve a high compression ratio; each scaling factor is stored using half-precision floating-point (FP16).

[0045] In the embodiments described above, Fisher information for each expert weight guides linear weighted fusion to extract shared base weights, improving the initial quality of quantization compression. The difference between each expert weight and the shared base weight is used as the residual weight for each expert, explicitly modeling individual differences among experts, fully preserving their individual expressive abilities, and preventing quantization performance degradation. A preset alternating refined binarization strategy is used to perform weight binarization on the shared base weights, optimizing the binarization quality and improving the performance retention of the compressed model. The residual weights of each expert are binarized layer by layer using a combination of sign matrix and scaling factor, further improving the compression ratio. Through an adaptive residual fitting binarization algorithm that includes a preset alternating refined binarization strategy and a combination of sign matrix and scaling factor, accuracy and compression rate are effectively balanced, improving the deployment efficiency of the hybrid expert large language model on edge devices. Furthermore, the entire process requires no retraining, achieving plug-and-play functionality, reducing deployment costs, and improving the feasibility of industrial applications.

[0046] To obtain Fisher information for each expert weight, in some specific embodiments of this application, for S12, the Fisher information for each expert weight is determined based on the activation value of each layer, which can be done using S121 to S124.

[0047] S121, construct the gradient approximation of each expert weight based on the input activation of each layer and the output activation of each expert.

[0048] Specifically as follows: Based on the forward propagation of the unquantized hybrid expert large language model, the input activation of each layer is recorded. And the output activation of each expert This is used to characterize the response of each layer to the input sample; By constructing a reference output That is, a static copy of the forward output, calculating the difference between the output activation of each expert and the reference output to approximate the effect of changes in expert weights on the model output; Activate based on the input of each layer Activate with the output of each expert With reference output The differences between them are used to construct the weights for each expert. W gradient approximation This describes the sensitivity of expert weights to the model output: in, This represents the gradient approximation of the expert weights. Indicates input activation. This indicates that the expert's output is activated. This indicates a static copy of the output.

[0049] S122, the element-wise square of the gradient approximation of each expert weight is used as the preliminary estimate of the weight sensitivity of each expert weight.

[0050] Specifically: in, This represents the preliminary estimate of the weight sensitivity of the expert weights. This represents the gradient approximation of the expert weights. This represents element-wise product.

[0051] S123, normalize the preliminary estimate of the weight sensitivity of each expert weight by using the overall magnitude of the input activation of each layer, and determine the weight sensitivity estimate result of each expert weight.

[0052] Specifically, in, This represents the weight sensitivity estimation result of the expert weights. This represents the preliminary estimate of the weight sensitivity of the expert weights. Indicates input activation. This represents the stability constant.

[0053] Specifically, the weight sensitivity estimation result F of the expert weights also represents the weight of Fisher information for each expert weight.

[0054] The weight sensitivity estimation results for each of the above expert weights are scaled consistently.

[0055] S124, average the weight sensitivity estimation results for each expert weight to determine the Fisher information for each expert weight.

[0056] Specifically, the preliminary estimation results of the weight sensitivity of each expert weight across multiple input samples or multiple forward batches. Accumulation and averaging are performed to form Fisher information for each expert weight, which is used to characterize the degree of influence of the expert weight on the model output.

[0057] In the embodiments described above, Fisher information of each expert and its corresponding layer weights is calculated based on the input activation value X and the expert output activation value Y of each layer, and the importance of these factors to the overall model performance is evaluated.

[0058] To obtain the shared basic weights, in some specific embodiments of this application, for S13, the shared basic weights are determined by linearly weighting and fusing each expert weight based on the Fisher information of each expert weight. This can be achieved by: W shared =F W exp in, This represents the weight of Fisher's information for each expert's weight. This represents the expert weight matrix arranged according to the expert dimension. "Indicates weighted operation based on expert dimensions" W shared This indicates that the basic weights are shared.

[0059] In the embodiments described above, Fisher information guides multiple expert weights to perform linear weighted fusion, extracting a high-quality shared weight structure, which is then used as a general backbone for unified storage, thereby improving the initial quality of 1-bit quantization compression.

[0060] To obtain the residual weight of each expert, in some specific embodiments of this application, for S14, the difference between each expert's weight and the shared basic weight is used as the residual weight of each expert to explicitly model the individual differences between experts. The residual weight retains the unique expressive ability of each expert and serves as a carrier of difference information, which can avoid the performance degradation problem caused by traditional pruning or merging methods. The residual weight will be further binarized to reduce storage overhead.

[0061] To guide subsequent quantization compression strategies, some specific embodiments of this application include a quantization method for a hybrid expert large language model, which further includes steps S16 to S17.

[0062] S16. Based on the Fisher information of each expert's weight, perform an importance analysis on the residual weight of each expert to determine the importance of the residual weight of each expert.

[0063] Specifically, based on the Fisher information of each expert weight determined in step S12 above, it is used as an importance index of the expert's residual weight to reflect the expert's relative contribution to the overall model.

[0064] The residual weights of each expert are weighted together with their corresponding Fisher information to highlight the more critical parts of each expert's residual weights.

[0065] Numerical analysis is performed on the weighted residual weights of the experts to measure their influence on the final model, thereby obtaining the importance distribution of the residual weights of each expert.

[0066] To avoid excessively large numerical ranges between different experts, Fisher information for each expert can be normalized, thereby making the importance assessment among experts more stable and comparable.

[0067] Based on the Fisher information of the experts after weighting, the importance of the residual weight of each expert is determined and used to guide the subsequent binarization and refinement of residual weights, retaining the more important residual weights and compressing the less important residual weights.

[0068] The importance of each expert's residual weight is used to characterize the criticality of that residual weight in maintaining model accuracy.

[0069] The importance of each expert's residual weight can be represented by an importance score.

[0070] S17. Based on the importance of the residual weights of each expert, the binarization order of the residual weights of each expert is assigned, and the binarization order of the residual weights of each expert is determined.

[0071] Specifically, the importance score of the residual weights of each expert is normalized to make them comparable across different levels / batches. The importance scores of the residual weights of each expert after normalization are then ranked. Based on a preset ratio, the residual weights of the experts with the highest ranking are taken as the residual weights of experts with higher importance, and the remaining portion is taken as the residual weights of experts with lower importance.

[0072] Different binarization orders are assigned to different experts. The residual weights of experts with higher importance retain more precision and are assigned a higher binarization order, while the residual weights of experts with lower importance retain less precision and are assigned a lower binarization order. This allows for a more aggressive compression strategy to be used for the residual weights of experts with lower importance.

[0073] Specifically, the binarization order represents the number of iterations in the "residual fitting binarization" process for the residual weights of the same expert. When the binarization order is 1, only one fitting of the "sign matrix and scaling factor" is performed to obtain a binary approximate representation of the residual weights of that expert. When the binarization order is 2, after completing the first fitting, the residuals are further calculated, and the "sign matrix and scaling factor" are refined again on the residuals, thereby improving the representation capability and accuracy by superimposing the two fittings.

[0074] In the specific implementation of this application, only two forms, first-order binarization and second-order binarization, are used to achieve a balance between accuracy and cost.

[0075] When the residual weights of experts are of high importance, they are assigned second-order binarization so that the residual weights of experts can be refined by two sign matrices and scaling factors, that is, by using two binarization approximations, which preserves higher expressive power. When the importance of an expert's residual weight is low, assign it first-order binarization so that the expert's residual weight can be refined by a first-order sign matrix and scaling factor, i.e., only one-order binarization approximation is used, thereby reducing additional computation and storage overhead.

[0076] In the embodiments described above, the importance of the residual weight of each expert is analyzed based on Fisher information of each expert weight, and the allocation of the binarization order is guided based on the importance of the residual weight of each expert, thereby guiding the subsequent quantization compression strategy and realizing hierarchical adaptive compression.

[0077] To compress weights using a hierarchical adaptive residual fitting binarization algorithm, in some specific embodiments of this application, for S15, a preset alternating refinement binarization strategy is used to perform weight binarization processing on the shared basic weights, which can be implemented as S151 to S157.

[0078] S151, perform initial binarization on the shared basic weights to generate the first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor.

[0079] Specifically, the shared basic weights are initially binarized to generate a first symbol matrix B1, and the initial first row scaling factor is obtained based on the row direction of the first symbol matrix B1. The initial scaling factor of the first column is obtained based on the column direction of the first symbol matrix B1. .

[0080] S152, Determine the initial binary approximation representation of the shared basic weights based on the first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor.

[0081] Specifically, the initial binary approximation of the shared basic weights is determined using the following formula: in, This indicates shared base weights. This indicates the scaling factor for the first row. This indicates the scaling factor for the second row. Represents the first symbol matrix, This represents a binary approximation of the shared basic weights.

[0082] Substitute the initial first row scaling factor, the initial second row scaling factor, and the first sign matrix into the above formula to obtain the initial binary approximate representation of the shared basic weights.

[0083] S153, construct the first residual term based on the reconstruction error between the shared basic weights and the initial binary approximation representation.

[0084] Specifically, the first residual term is constructed as follows: in, 1 represents the first residual term, which is used to characterize the deviation of the current binarized approximation of the shared basic weights from the original shared basic weights.

[0085] S154. In the next round of alternating refining, the first residual term of the previous round of alternating refining is used as the optimization target. A greedy parameter update strategy is adopted to select the update direction with the largest decrease in the first residual term. The first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor are updated to determine the new first symbol matrix, the new first row scaling factor, and the new first column scaling factor.

[0086] Specifically, the direction of the first sign matrix is ​​updated according to the sign direction of the first residual term, thereby determining the new first sign matrix: The scaling factor of the first row is updated based on the correspondence between the shared basic weights and the current first symbol matrix B1, thereby determining the new scaling factor of the first row: The scaling factor of the first column is updated based on the statistical information of the column direction sharing the basic weights, thereby determining the new scaling factor of the first column: S155, in the next round of alternating refinement, a new binary approximation representation of the shared basic weights is determined based on the new first symbol matrix, the new first row scaling factor, and the new first column scaling factor.

[0087] Specifically, the process of step S155 can be referred to step S152 above, and will not be repeated here.

[0088] S156, In the next round of alternating refinement, a new first residual term is constructed based on the reconstruction error between the shared basic weights and the new binary approximation.

[0089] Specifically, the process of step S156 can be referred to step S153 above, and will not be repeated here.

[0090] S157, repeat the alternating refining process multiple times until the difference between the new binarized approximate representations of the shared basic weights in two adjacent alternating refining processes is less than the preset first critical value or reaches the preset first iteration round, and determine the binarization result of the shared basic weights.

[0091] Specifically, steps S154 to S156 are repeated cyclically to perform multiple rounds of alternating refining processes.

[0092] In the embodiments described above, a preset alternating refinement binarization strategy is adopted. Through multiple rounds of alternating refinement, the shared basic weights gradually approach their binarized representation. Finally, based on the first row scaling factor, the first column scaling factor, and the first symbol matrix, the shared basic weights W obtain high-quality binarization results without training, and better maintain the weight structure characteristics in the column direction. Furthermore, in each round of alternating refinement, a greedy parameter update strategy is adopted to select the update direction with the largest decrease in the current first residual term, and the first row scaling factor, the first column scaling factor, and the first symbol matrix are preferentially adjusted. This ensures that each iteration can minimize the main sources of error and improve the stability and speed of binarization convergence.

[0093] To achieve weight binarization of the residual weights of each expert layer by layer, in some specific embodiments of this application, for S15, the residual weights of each expert are binarized by combining a sign matrix and a scaling factor, which can be S158 to S164.

[0094] S158 performs initial binarization on the residual weights of each expert to generate a second symbol matrix, an initial second row scaling factor, and an initial second column scaling factor.

[0095] Specifically, the residual weight R of each expert is initially binarized to generate a second symbol matrix B2, and the initial second row scaling factor is obtained based on the row direction of the second symbol matrix B2. And obtain the initial second column scaling factor based on the column direction of the second symbol matrix B2. .

[0096] S159, based on the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor, determine the initial binary approximate representation of the residual weights for each expert.

[0097] Specifically, the initial binary approximation of the expert's residual weights is determined using the following formula: in, This represents the residual weights of the experts. This indicates the scaling factor for the second row. This indicates the scaling factor in the second column. Represents the second symbol matrix, This represents the initial binary approximation of the residual weights of the experts.

[0098] Substitute the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor into the above formula to obtain the initial binary approximate representation of the expert's residual weights.

[0099] S160, construct the second residual term based on the reconstruction error between the residual weights of each expert and the initial binary approximation representation of the residual weights of each expert.

[0100] Specifically, the second residual term is constructed as follows: in, The second residual term represents the binary approximation of the residual weights of the current expert, which deviates from the residual weights of the original expert.

[0101] S161, In the next round of alternating refining, the second residual term of the previous round of alternating refining is used as the optimization objective. A greedy parameter update strategy is adopted to select the update direction with the largest decrease in the second residual term, update the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor, and determine the new second symbol matrix, the new second row scaling factor, and the new second column scaling factor.

[0102] Specifically, the direction of the second sign matrix is ​​updated according to the sign direction of the second residual term, thereby determining the new second sign matrix: The second row scaling factor is updated based on the correspondence between the expert's residual weights and the current second sign matrix, thereby determining the new second row scaling factor: The scaling factor of the second column is updated based on the correspondence between the expert's residual weights and the current second sign matrix, thereby determining the new scaling factor of the second column: S162, in the next round of alternating refinement, a new binary approximation of the residual weights of each expert is determined based on the new second symbol matrix, the new second row scaling factor, and the new second column scaling factor.

[0103] Specifically, in, This represents a new binary approximation.

[0104] S163, In the next round of alternating refinement, a new second residual term is constructed based on the reconstruction error between the residual weights of each expert and the new binary approximate representation of the residual weights of each expert.

[0105] Specifically, the process of step S163 can be referred to step S160 above, and will not be repeated here.

[0106] S164, repeat the alternating refinement process multiple times until the difference between the new binarized approximate representations of the residual weights of each expert in two adjacent alternating refinement processes is less than the preset second critical value or the preset second iteration is reached, and determine the binarization result of the residual weights of each expert.

[0107] Specifically, the number of cycles in the alternating refinement process of the residual weights of experts in this embodiment is determined based on the binarization order of the residual weights of each expert determined in step S17 above, and steps S161 to S163 above are executed cyclically to perform multiple rounds of alternating refinement process.

[0108] In step S15 of this application, a preset alternating refinement binarization strategy is used to perform weight binarization on the residual weights of each expert. Essentially, this involves using a combination of a sign matrix and a scaling factor to perform weight binarization on the residual weights of each expert. The shared basic weights of experts have high performance requirements, so multiple rounds of alternating refinement are used. The residual weights of experts are generally low-rank matrices, so multiple rounds of alternating refinement are not required to maintain strong performance. Therefore, the combination of a sign matrix and a scaling factor is used to reduce compression overhead.

[0109] In the embodiments described above, a combination of a symbol matrix and a scaling factor is used to perform weight binarization on the residual weights of each expert. The residual weights of the experts gradually approximate their binarized representations. Finally, based on a small number of alternating refinements using the scaling factor in the second row, the scaling factor in the second column, and the second symbol matrix, the residual weights of the experts achieve high-quality binarization results without training, while effectively preserving the important parameters personalized by each expert. Furthermore, in each round of alternating refinement, a greedy parameter update strategy is used to select the update direction that can minimize the quantization error under the current second residual term, and the update results are gradually corrected so that each iteration can minimize the main sources of error to the greatest extent, thereby improving the stability and convergence speed of the binarization approximation.

[0110] This application employs an adaptive residual fitting binarization algorithm that combines a preset alternating refinement binarization strategy with a combination of symbol matrix and scaling factor to compress weights layer by layer. For each expert and its hierarchical structure, hierarchical optimization and binarization operations are performed as needed to ensure that the compression process takes into account model accuracy, compression ratio and hardware friendliness, effectively balances accuracy and compression rate, and improves the deployment efficiency of hybrid expert large language models on edge devices.

[0111] This application presents a quantization method for hybrid expert large language models, proposing a novel residual compression framework that breaks through the traditional binary paradigm of "pruning or merging." It adopts a modeling approach of "shared basic weights + personalized difference weights" to effectively separate and compress common and individual information of experts. Fisher information guides expert fusion to extract shared basic weights, and residual weights are constructed to represent the differences for each expert. Subsequently, an adaptive residual fitting binarization algorithm is used to quantize the shared basic weights and residual weights to 1 bit. The entire process requires no additional fine-tuning or retraining, exhibiting high compression ratio, low computational cost, and good performance preservation. It also boasts plug-and-play deployment advantages, reducing deployment costs and making it suitable for efficient deployment scenarios of large-scale hybrid expert large language models, thus improving the feasibility of industrial applications.

[0112] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0113] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0114] Example 1: The quantization method of the hybrid expert large language model provided in this application is used to perform 1-bit quantization compression and verification on the hybrid expert model based on Mixtral-8×7B.

[0115] First, 2K high-quality cue data points are selected as calibration samples from the Mixtral-8×7B-based hybrid expert model. The calibration samples are then input into the Mixtral-8×7B-based hybrid expert model for forward propagation to extract the intermediate activations of each expert and calculate the Fisher information matrix. After calculating the sensitivity score of each expert based on the Fisher information, a linear weighted fusion operation is performed on the expert weights to obtain the shared basic weight matrix.

[0116] Experimental results show that dynamic linear weighted fusion of basic weights can significantly reduce redundant subspace overlap among experts and improve subsequent quantization adaptability.

[0117] Subsequently, based on the difference between each expert's weight and the shared basic weight, a corresponding residual weight is constructed for each expert. Then, based on the Fisher information of each expert's weight, importance analysis and binarization order assignment are performed on the residual weight of each expert.

[0118] Finally, for expert residual weights with high sensitivity, a combination of retaining the scaling factor and the sign matrix is ​​used to enhance expressive power; for expert residual weights with low sensitivity, a strict 1-bit compression strategy is adopted to improve the overall compression rate.

[0119] In a specific implementation, for expert residual weights with low sensitivity, a strict first-order binary compression strategy can be adopted to further improve the overall compression rate, namely, a combination of scaling factor and sign matrix. Under this strategy, a preset alternating refined binary strategy is first used to compress the shared basic weights, and then a combination of scaling factor and sign matrix is ​​used to perform 1-bit compressed representation of the expert residual weights.

[0120] In the first-order binary compression strategy, each channel retains only the sign matrix to represent the positive and negative directions of the weights, and a channel-level scaling factor is introduced to maintain the basic amplitude relationship. The scaling factor is small and can be approximated as a small amount, which will not significantly increase the storage or computational overhead. Therefore, the whole can still be regarded as a 1-bit compressed form.

[0121] For these low-sensitivity expert residual weights, compression can be achieved with only one sign extraction and one scaling factor generation, without the need for multiple rounds of alternating refinement. In other words, this part of the weights can be represented using a simple first-order binarization form, thereby maximizing the overall compression ratio of the model.

[0122] Without any training, the quantization method for the hybrid expert large language model provided in this application can reduce the total memory usage of the model to about 1 / 20 of the original model, while only improving the perplexity of WikiText-2 by about 1.5–2.1, thus maintaining a high level of language modeling capability. Example 2: Hierarchical adaptive residual quantization is performed on the DeepSeek-MoE architecture using a quantization method based on a hybrid expert large language model provided in this application.

[0123] Specifically, this embodiment adopts a DeepSeek-style 16-expert MoE architecture.

[0124] First, hierarchical Fisher information evaluation is performed on all experts in the DeepSeek-MoE architecture, and the importance of feature dimensions at different levels is ranked separately. In multi-head feedforward networks, since the output layer weights have a significant impact on the final prediction, the Fisher sensitivity is significantly higher than that of the inner layers. Therefore, a higher-order binarization form (including a learnable scaling factor) is assigned to this part of the residual.

[0125] In skip connections and low-rank paths within expert systems, the sensitivity is relatively low, and the compression strategy employs a pure symbolic matrix representation to maximize the compression ratio.

[0126] Specific deployment results show that, without any parameter backhaul training, the method provided in this application reduces the number of model parameters to about 8%–10% of the original, reduces inference latency by an average of 38%, and maintains an average accuracy decrease of less than 1% on multi-task evaluations (including MMLU, ARC-C, and HellaSwag).

[0127] Example 3: A text generation method based on a hybrid expert large language model, comprising: Step 1: Obtain natural language prompts, context sequences, instruction text, historical dialogues, and the text generation model to be quantized.

[0128] For example, the text generation model can adopt the DeepSeekMoE-16B-Base model.

[0129] Step 2: The text generation model to be quantized is quantized using the quantization method of the hybrid expert large language model provided in this application, and the quantized text generation model is determined. Step 3: Input the natural language prompts, context sequence, instruction text, and historical dialogue into the quantified text generation model to determine the text generation result.

[0130] The embodiments described above employ a quantization method for a hybrid expert large language model provided in this application to quantize the text generation model. An adaptive residual fitting binarization algorithm, incorporating a preset alternating refinement binarization strategy and a combination of a symbol matrix and scaling factor, is used to compress weights for each expert and its hierarchical structure. Hierarchical optimization and binarization operations are performed as needed, ensuring that the compression process balances model accuracy, compression ratio, and hardware friendliness. This effectively balances accuracy and compression rate, improves the performance of the text generation task, and reduces storage and computational overhead.

[0131] Example 4: A knowledge extraction method based on a hybrid expert large language model, comprising: Step 1: Obtain the text to be analyzed, factual questions, entity relationship queries, retrieval prompts, contextual text, and knowledge extraction models.

[0132] For example, the knowledge extraction model can adopt DeepSeekMoE-16B-Base.

[0133] Step 2: The knowledge extraction model is quantified using the quantification method of the hybrid expert large language model provided in this application to determine the quantified knowledge extraction model.

[0134] Step 3: Input the text to be analyzed, factual questions, entity relationship queries, retrieval prompts, and contextual text into the quantified knowledge extraction model to determine the knowledge to be extracted.

[0135] The embodiments described above employ a quantization method for a hybrid expert large language model provided in this application to quantize the knowledge extraction model. An adaptive residual fitting binarization algorithm, incorporating a preset alternating refinement binarization strategy and a combination of symbolic matrix and scaling factor, is used to compress weights for each expert and its hierarchical structure. Hierarchical optimization and binarization operations are performed as needed, ensuring that the compression process balances model accuracy, compression ratio, and hardware friendliness. This effectively balances accuracy and compression rate, improves the performance of the knowledge extraction task, and reduces storage and computational overhead.

[0136] Figure 2 This is a schematic diagram illustrating the structure of a quantization system for a hybrid expert large language model according to an exemplary embodiment.

[0137] Reference Figure 2 As shown in one embodiment of this application, a quantization system 100 for a hybrid expert large language model is provided, including: an activation value acquisition module 110, a Fisher information extraction module 120, a shared basic weight determination module 130, a residual weight determination module 140, and a weight binarization processing module 150.

[0138] The activation value acquisition module 110 is used to input a preset number of input samples into the hybrid expert large language model to be quantized, perform forward propagation, and determine the activation value of each layer of the hybrid expert large language model to be quantized. Fisher information extraction module 120 is used to determine Fisher information for each expert weight based on the activation value of each layer. The shared basic weight determination module 130 is used to perform linear weighted fusion on each expert weight based on the Fisher information of each expert weight to determine the shared basic weight. The residual weight determination module 140 is used to take the difference between each expert's weight and the shared basic weight as the residual weight of each expert. The weight binarization module 150 is used to perform weight binarization on the shared basic weights using a preset alternating refinement binarization strategy, and to perform weight binarization on the residual weights of each expert using a combination of symbol matrix and scaling factor, so as to determine the quantized hybrid expert large language model.

[0139] In the embodiments described above, Fisher information for each expert weight guides linear weighted fusion to extract shared base weights, improving the initial quality of quantization compression. The difference between each expert weight and the shared base weight is used as the residual weight for each expert, explicitly modeling individual differences among experts, fully preserving their individual expressive abilities, and preventing quantization performance degradation. A preset alternating refined binarization strategy is used to perform weight binarization on the shared base weights, optimizing the binarization quality and improving the performance retention of the compressed model. The residual weights of each expert are binarized layer by layer using a combination of sign matrix and scaling factor, further improving the compression ratio. Through an adaptive residual fitting binarization algorithm that includes a preset alternating refined binarization strategy and a combination of sign matrix and scaling factor, accuracy and compression rate are effectively balanced, improving the deployment efficiency of the hybrid expert large language model on edge devices. Furthermore, the entire process requires no retraining, achieving plug-and-play functionality, reducing deployment costs, and improving the feasibility of industrial applications.

[0140] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0141] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0142] Based on the same technical concept, in some specific embodiments of this application, a computer-readable storage medium is provided on which a computer program is stored, which can be used to execute a method when the program is executed by a processor.

[0143] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0144] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0145] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0146] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0147] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A quantization method for a hybrid expert large language model, characterized in that, include: A preset number of input samples are input into the hybrid expert large language model to be quantized, and forward propagation is performed to determine the activation value of each layer of the hybrid expert large language model to be quantized. Based on the activation value of each layer, determine the Fisher information for each expert weight; Based on the Fisher information of each expert weight, the expert weights are linearly weighted and fused to determine the shared basic weights. The difference between each expert's weight and the shared basic weight is used as the residual weight of each expert. The shared basic weights are binarized using a preset alternating refinement binarization strategy, and the residual weights of each expert are binarized using a combination of symbol matrix and scaling factor to determine the quantized hybrid expert large language model.

2. The quantization method for a hybrid expert large language model according to claim 1, characterized in that, The method further includes: Based on the Fisher information of each expert's weight, an importance analysis is performed on the residual weight of each expert to determine the importance of the residual weight of each expert. Based on the importance of the residual weights of each expert, the binarization order of the residual weights of each expert is assigned, thereby determining the binarization order of the residual weights of each expert.

3. The quantization method for a hybrid expert large language model according to claim 1, characterized in that, The step of determining the Fisher information for each expert weight based on the activation value of each layer includes: Based on the input activation of each layer and the output activation of each expert, a gradient approximation of the weights of each expert is constructed. The element-wise square of the gradient of each expert weight is used as a preliminary estimate of the weight sensitivity of each expert weight. The overall magnitude of the input activation at each layer is used to normalize the preliminary estimate of the weight sensitivity of each expert weight, thereby determining the weight sensitivity estimation result of each expert weight. The Fisher information for each expert weight is determined by averaging the weight sensitivity estimation results for each expert weight.

4. The quantization method for a hybrid expert large language model according to claim 1, characterized in that, The step of performing weight binarization on the shared basic weights using a preset alternating refinement binarization strategy includes: The shared basic weights are initially binarized to generate a first symbol matrix, an initial first row scaling factor, and an initial first column scaling factor. Based on the first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor, determine the initial binary approximate representation of the shared basic weights; Based on the reconstruction error between the shared basic weights and the initial binary approximation representation, a first residual term is constructed; In the next round of alternating refining, the first residual term from the previous round of alternating refining is used as the optimization target. A greedy parameter update strategy is adopted to select the update direction with the largest decrease in the first residual term, and the first symbol matrix, the initial first row scaling factor, and the initial first column scaling factor are updated to determine the new first symbol matrix, the new first row scaling factor, and the new first column scaling factor. In the next round of alternating refining, a new binarized approximate representation of the shared basic weights is determined based on the new first symbol matrix, the new first row scaling factor, and the new first column scaling factor. In the next round of alternating refining, a new first residual term is constructed based on the reconstruction error between the shared basic weights and the new binary approximation representation; The process involves multiple rounds of alternating refinement until the difference between the new binarized approximate representations of the shared basic weights in two adjacent rounds of alternating refinement is less than a preset first critical value or reaches a preset first iteration round, at which point the binarization result of the shared basic weights is determined.

5. The quantization method for a hybrid expert large language model according to claim 2, characterized in that, The method of weight binarization of the residual weights of each expert using a combination of a sign matrix and a scaling factor includes: The residual weights of each expert are initially binarized to generate a second symbol matrix, an initial second row scaling factor, and an initial second column scaling factor. Based on the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor, determine the initial binarized approximate representation of the residual weights of each expert; A second residual term is constructed based on the reconstruction error between the residual weight of each expert and the initial binary approximation representation of the residual weight of each expert. In the next round of alternating refining, the second residual term from the previous round of alternating refining is used as the optimization target. A greedy parameter update strategy is adopted to select the update direction with the largest decrease in the second residual term, and the second symbol matrix, the initial second row scaling factor, and the initial second column scaling factor are updated to determine the new second symbol matrix, the new second row scaling factor, and the new second column scaling factor. In the next round of alternating refining, a new binarized approximate representation of the residual weights of each expert is determined based on the new second symbol matrix, the new second row scaling factor, and the new second column scaling factor. In the next round of alternating refining, a new second residual term is constructed based on the reconstruction error between the residual weight of each expert and the new binarized approximate representation of the residual weight of each expert. The process involves multiple rounds of alternating refinement until the difference between the new binarized approximate representations of the residual weights of each expert in two adjacent rounds of alternating refinement is less than a preset second critical value or a preset second iteration is reached. The binarization result of the residual weights of each expert is then determined.

6. A quantization system for a hybrid expert large language model, characterized in that, include: The activation value acquisition module is used to input a preset number of input samples into the hybrid expert large language model to be quantized, perform forward propagation, and determine the activation value of each layer of the hybrid expert large language model to be quantized. The Fisher information extraction module is used to determine the Fisher information for each expert weight based on the activation value of each layer. The shared basic weight determination module is used to perform linear weighted fusion on each expert weight based on the Fisher information of each expert weight to determine the shared basic weight. The residual weight determination module is used to take the difference between each expert weight and the shared basic weight as the residual weight of each expert. The weight binarization module is used to perform weight binarization on the shared basic weights using a preset alternating refinement binarization strategy, and to perform weight binarization on the residual weights of each expert using a combination of symbol matrix and scaling factor, so as to determine the quantized hybrid expert large language model.

7. A text generation method based on a hybrid expert large language model, characterized in that, include: Acquire natural language prompts, context sequences, instruction text, historical dialogues, and a text generation model to be quantized; The text generation model to be quantized is quantized using the quantization method of the hybrid expert large language model according to any one of claims 1-5, and the quantized text generation model is determined. The natural language prompts, the context sequence, the instruction text, and the historical dialogue are input into the quantized text generation model to determine the text generation result.

8. A knowledge extraction method based on a hybrid expert large language model, characterized in that, include: Obtain the text to be analyzed, factual questions, entity relationship queries, retrieval prompts, contextual text, and knowledge extraction models; The knowledge extraction model is quantified using the quantization method of the hybrid expert big language model described in any one of claims 1-5 to determine the quantified knowledge extraction model. The text to be analyzed, the factual question, the entity relationship query, the retrieval prompt, and the context text are input into the quantified knowledge extraction model to determine the extracted knowledge.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-8.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-8.

Images