Hybrid expert model-oriented reasoning acceleration and performance compensation method and system

Abstract

The invention relates to the technical field of data processing, and particularly discloses a reasoning acceleration and performance compensation method and system for a hybrid expert model.The method comprises the steps that on the premise that the performance loss amount of a model to be optimized is smaller than a performance loss threshold value, a plurality of MoE layers of the model to be optimized are selected, and the number of experts activated by the MoE layers is reduced; a MoE layer sensitive to the number of experts is selected, and a Hook module is added; the Hook module is located in front of the expert module of the MoE layer; the Hook module adopts a two-layer linear transformation and nonlinear activation residual structure; and training the Hook module to obtain an optimized model. According to the method, the number of activated experts can be greatly reduced in the MoE reasoning process, and the performance loss can be compensated through a lightweight means, so that an efficient, deployable and migratable practical MoE reasoning acceleration framework is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for accelerating inference and compensating performance for hybrid expert models. Background Technology

[0002] The Mixture-of-Experts (MoE) architecture, as an important paradigm for scaling large models, decouples parameter count from computational cost by deploying multiple expert networks in each MoE layer and using a gating mechanism to route each token to the top-k highest-scoring experts. This significantly increases model capacity while requiring only a small number of experts to be activated at each inference step. This activation sparsity significantly reduces the computational cost per unit of inference, making it possible to use larger-capacity models with limited computing resources.

[0003] Meanwhile, the healthcare field is rapidly seeing the emergence of multimodal medical analysis models with "visual information such as images / pathology / endoscopy + textual information such as electronic medical records / examination and testing / guideline evidence" as core inputs. These models often need to complete cross-modal understanding, evidence alignment, and structured generation / reasoning within a single consultation or hospital workflow, placing more stringent demands on inference latency, throughput, and deployability than in general-purpose scenarios. Especially in scenarios such as private deployment within hospitals, edge device-assisted image reading, and rapid decision-making via emergency / ICU bypass, limited computing resources and sensitivity to real-time interactive responses make reducing inference overhead without significantly impacting the effectiveness of medical tasks a crucial prerequisite for the successful implementation of large-scale multimodal medical model projects.

[0004] To enhance the capacity and cross-modal reasoning capabilities of multimodal medical models, some systems introduce hybrid expert structures into the language backbone or cross-modal fusion feedforward modules, or employ a combination of general experts and medical domain / task experts to adapt to different organ systems, imaging modalities, and clinical subtasks. In this case, the Top-k expert activation strategy in the MoE inference stage also incurs significant computational and scheduling overhead; however, medical scenarios place high emphasis on output consistency and security, making simplistic Top-k downscaling often unacceptable.

[0005] Existing MoE models suffer from several shortcomings: a lack of effective performance compensation mechanisms when significantly reducing the number of activated experts; inability to achieve transferable and pluggable compensation under constraints of minimal calibration samples and minimal parameter modifications; a lack of systematic analysis of the sensitivity of MoE models to the number of experts from the perspective of layer depth distribution, failing to fully explore the potential for fewer experts in upper layers; and insufficient general support for different MoE architectures (coarse-grained experts, fine-grained experts, shared expert structures, etc.). Furthermore, in multimodal medical analysis models, existing acceleration solutions largely focus on the quantization, pruning, and distillation of the visual encoder or language backbone, paying insufficient attention to the activation sparsity of cross-modal fusion stages or medical expert routing structures. Especially when the model employs a MoE or hybrid structure with multiple types of experts (general / medical / organ system / task), there is a lack of a lightweight method that can proactively compensate for cross-modal semantic alignment offsets caused by reducing experts under conditions of minimal cross-modal calibration samples. This deficiency makes it difficult for existing technologies to simultaneously meet the comprehensive requirements of "accelerated inference, consistency in diagnosis / triage, pluggable deployment, and rapid migration" in medical scenarios. Summary of the Invention

[0006] The present invention aims to solve the above-mentioned problems. To this end, the present invention provides a method and system for inference acceleration and performance compensation for hybrid expert models, which can significantly reduce the number of activated experts during MoE inference and compensate for performance loss through lightweight means, so as to realize an efficient, deployable, and transferable practical MoE inference acceleration framework for multimodal medical analysis.

[0007] This invention provides a method for accelerating inference and compensating performance for hybrid expert models, and the technical solution adopted is as follows: including: S1: Obtain the model to be optimized; S2: Provided that the performance loss of the model to be optimized is less than the performance loss threshold, select several MoE layers of the model to be optimized to reduce the number of activated experts. S3: Select the MoE layer which is more sensitive to the number of experts and add a Hook module; the Hook module is located before the expert module of the MoE layer; the Hook module adopts a residual structure with two layers of linear transformation and nonlinear activation; S4: Train the Hook module to obtain an optimized model, and use the optimized model for inference.

[0008] Furthermore, the model to be optimized is a pre-trained multimodal medical analysis model.

[0009] Furthermore, the model to be optimized includes multiple MoE layers, each MoE layer containing multiple expert modules, and the expert modules are feedforward neural networks.

[0010] Furthermore, in step S3, the process traverses from the last MoE layer to the first MoE layer; the number of activated experts is reduced in the traversed MoE layers, and the performance loss of the model to be optimized is evaluated; the first M MoE layers with performance loss not less than the performance degradation threshold are selected and added to the Hook module.

[0011] Furthermore, M is 1.

[0012] Furthermore, the Hook module hides the input state. The processing method is as follows: in, This is the first trainable linear layer. The second trainable linear layer, It is a non-linear activation function. Scalar scaling factor This refers to the output characteristics of the Hook module.

[0013] Furthermore, in step S4, the model to be optimized is frozen, and only the Hook module and the scalar scaling factor are updated.

[0014] Furthermore, in step S2, the expert activation strategy is to activate the top-k experts with the highest probability of activating the route output; The Lth layer to the last MoE layer of the model to be optimized is set to use kn experts, while the remaining MoE layers keep k experts. When L takes different values, the performance loss of the model to be optimized is tested respectively, and the L value with the largest performance loss less than the performance loss threshold and the largest reduction in the number of activated experts is obtained.

[0015] Furthermore, the number of activation experts in the MoE layer is reduced from 2 to 1.

[0016] This invention also provides a reasoning acceleration and performance compensation system for hybrid expert models, the technical solution of which is as follows: including: The expert number adjustment module is used to obtain the model to be optimized; provided that the performance loss of the model to be optimized is less than the performance loss threshold, select several MoE layers of the model to be optimized and reduce the number of activated experts. The Hook module is used to select MoE layers that are sensitive to the number of experts and add Hook modules to them; the Hook module is located before the expert modules of the MoE layer; the Hook module adopts a residual structure with two layers of linear transformation and nonlinear activation. The model training module is used to train the Hook module to obtain the optimized model.

[0017] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. This invention can maintain model performance while significantly reducing the number of activated experts. It reduces the number of activated experts by 50% or even 75% while ensuring that the model's performance degradation in general language understanding, multi-task reasoning, and mathematical reasoning tasks does not exceed the performance loss threshold, thus significantly reducing the computational load of MoE inference.

[0018] 2. This invention implements a lightweight, pluggable performance compensation mechanism. By inserting a Hook module with extremely small parameters into a selected Transformer layer, and only fine-tuning this Hook module, large-scale updates to the original MoE experts and backbone network are avoided, making this invention easily deployable on various existing open-source MoE models.

[0019] 3. Under multimodal input conditions such as medical images-text, pathology-report, and laboratory indicators-medical records, this invention significantly reduces the number of experts activated by MoE while maintaining consistency between cross-modal semantic alignment and medical task output. It also supports rapid compensation training using a very small amount of cross-modal calibration data to meet the transferability requirements under private deployment within hospitals and different hospital / equipment / population distribution differences.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the method provided by the present invention.

[0023] Figure 2 This is a diagram of the Hook module compensation architecture provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0025] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0026] The following is combined Figure 1 and Figure 2 The present invention will be further described in detail below, providing a method and system for inference acceleration and performance compensation for hybrid expert models: In this embodiment, as Figure 1 As shown, a method for inference acceleration and performance compensation for hybrid expert models is provided, including the following steps: S1: Obtain the model to be optimized.

[0027] The model to be optimized is a trained model, such as a MoE large language model containing multiple Transformer blocks. The model contains MoE substructures, that is, the model to be optimized includes multiple MoE layers, each of which contains multiple expert modules, and the expert modules are feedforward neural networks.

[0028] S2: Provided that the performance loss of the model to be optimized is less than the performance loss threshold, select several MoE layers of the model to be optimized to reduce the number of activated experts.

[0029] This method first conducts a systematic analysis of the MoE model: the expert activation strategy of the MoE layer is to activate the top-k experts with the highest probability of routing output. While keeping other conditions constant, the top-k threshold is adjusted in different MoE layer intervals; for example, the top-k threshold of the MoE layer is adjusted from 2 to 1, and the impact on the average performance of multiple general tasks is observed. The systematic analysis shows that the model is significantly less sensitive to the number of experts in the middle and upper layers (especially in several layers near the output end) than in the lower layers: when the top-2 is reduced to top-1 in deeper layers, the model's performance on the general benchmark decreases by about 1%. Accuracy is commonly used to evaluate model performance. For different types of inference tasks, multiple indicators can also be selected for comprehensive evaluation based on the specific inference task type.

[0030] Based on this, this method proposes the Top Layer Needs Less Experts (TLNLE) strategy: the top layer of the model is regarded as a region that is insensitive to the number of experts, and a smaller Top-k is uniformly adopted in this region; the original Top-k is retained in several layers near the bottom to ensure the model's basic semantic modeling ability for the input token; by searching the index of the starting layer, the layer split point with the smallest activation average number of experts is found within the set performance loss threshold (e.g., 2.5%).

[0031] In this embodiment, the range of MoE layers with reduced activation experts is determined as follows: the Lth layer to the last MoE layer of the model to be optimized is set to use kn experts, while the remaining MoE layers maintain k experts; the performance loss of the model to be optimized is tested for different values ​​of L; the value of L that results in the largest reduction in the number of activation experts while the performance loss is less than the performance loss threshold is obtained. L is used as the layer split point. In this embodiment, when k=2 and n=1, the number of activation experts in the MoE layer is reduced from 2 to 1.

[0032] This strategy can be used as a dynamic -k routing strategy for layer-depth awareness, reducing the number of activation experts without introducing additional structures and training overhead, thus freeing up more acceleration space for subsequent Hook modules.

[0033] S3: Select the MoE layer that is more sensitive to the number of experts and add a Hook module; the Hook module is located before the expert module in the MoE layer.

[0034] After employing TLNLE and Top-k reduction strategies, the model can still maintain acceptable performance, but a performance loss of several percentage points still occurs. Therefore, this embodiment designs a pluggable, lightweight Hook module and inserts it before the feed-forward network of the selected Transformer layer. In this embodiment, the Hook module adopts a two-layer linear transformation plus nonlinear activation residual structure, which hides the input state... The processing method is as follows: in, The output characteristics of the Hook module, This is the first trainable linear layer. The second trainable linear layer, It is a non-linear activation function; This is a scalar scaling factor, initially set to 0.5, and updated during training. For coarse-grained expert models, the intermediate dimension of the Hook module is preferably set to the intermediate dimension of the expert modules in MoE. For fine-grained expert models, the intermediate dimension of the Hook module is preferably set to twice the intermediate dimension of the expert modules in MoE to enhance compensation capabilities.

[0035] This method, based on TLNLE performance analysis, selects one or more MoE layers that are sensitive to the number of experts and exhibit relatively significant performance degradation as insertion layers for the Hook module. Specifically, it iterates from the last MoE layer to the first MoE layer; the number of activated experts is reduced in each traversed MoE layer, and the performance loss of the model to be optimized is evaluated; the first M MoE layers with performance loss not less than the performance degradation threshold are selected to add the Hook module. Preferably, only one MoE layer is selected to add the Hook module, i.e., M is 1, and the first MoE layer with performance degradation not less than the performance degradation threshold is selected to add the Hook module. In this embodiment, the performance degradation threshold is 1%. This MoE layer is usually located in the upper middle part of the model.

[0036] This method reduces the performance loss caused by the number of activated experts through layer-by-layer evaluation, selecting the layer with the optimal performance-parameter trade-off as the insertion layer for the Hook module. Before inserting the Hook module into the feedforward network of this MoE layer, the input... Replace with the output of the Hook module The data is then fed into the MoE for routing and expert calculation. The Hook module only performs state-space correction on the representation space of a specific layer's input, without changing the internal structure and routing mechanism of the MoE layer, thus exhibiting good pluggability and compatibility.

[0037] S4: Train the Hook module to obtain an optimized model, and use the optimized model for inference.

[0038] In this embodiment, a small-scale unlabeled calibration dataset is used to train the Hook module to approximate the difference in the intermediate state space between the original Top-2 model and the model after reducing the number of experts, thereby compensating for the difference.

[0039] For general task scenarios, a small number of samples, such as 512, are randomly sampled from general corpora (e.g., web page text, natural language sentence sets), and the sequence length of each sample can be set to 2048. For specific domain tasks such as mathematical reasoning, samples of the same size are sampled from the corresponding domain dataset (e.g., mathematical problems and their solution texts).

[0040] Training objectives and processes, let S be the training goal and process. oSc represents the state space of the original Top-2 model at a certain layer; Sc represents the state space of the model after reducing the number of activated experts (e.g., Top-1); SH represents the state space of the model after adding the Hook module. This embodiment minimizes the difference between the Hook module output and the target state space, allowing SH to compensate as much as possible for Sc relative to S. o The offset is adjusted to reduce overall performance loss. For example... Figure 2 As shown, during training: the model to be optimized is frozen, and only the Hook module and scalar scaling factor are updated. A standard optimizer, such as Adam, is used; the learning rate can be set to 1×10. -4 Iterate through several rounds, such as 1000 epochs or an equivalent number of update steps; judge the training convergence by combining the loss decrease trend on the calibration set with the performance change on a small evaluation set.

[0041] Experiments show that the number of parameters in the Hook module typically accounts for only 0.14%–0.70% of the total model parameters. Training can be completed in a few hours on an 8-GPU high-performance environment, with a cost far less than fine-tuning the entire model or large-scale PEFT methods.

[0042] During inference, the router selects experts according to the Top-k+TLNLE rules determined by this method. For the MoE layer with the Hook module inserted, the Hook calculation is performed before being sent to the MoE and feedforward network. By reducing the number of activated experts, the computational load of operators such as matrix multiplication is directly reduced, while the overhead of cross-expert calls and scheduling is reduced, thereby shortening the Time-To-First-Token (TTFT) in the prefill stage and the Inter-Token Latency (ITL) in the decoding stage. Based on the optimization of models such as Mixtral-8×7B, this invention can reduce the activation parameters by 50%–75% while maintaining an average performance decrease of about 2.5%, and achieves an inference acceleration of about 1.42 times in the prefill stage on different GPU hardware platforms, as shown in Table 1.

[0043] Table 1

[0044] This embodiment also evaluated the performance of the proposed method and various typical MoE models on multiple datasets (ARC-c, ARC-e, RTE, WinoGrande, etc.). Typical MoE models include Mixtral-8×7B and Mixtral-8×22B, and the results are shown in Table 2.

[0045] Table 2 In Table 2, Act. represents the average number of activated experts per token. The evaluation metric is the average score on the evaluation set, calculated as the number of correct answers divided by the total number of experts. Reducing the number of experts directly from Top-2 to Top-1 would result in an average performance decrease of approximately 5 percentage points for general tasks. Using this method, with the activation parameters reduced to approximately 57%, the average task performance can be recovered by 2.4%-3.0%, with the final performance decreasing by only about 2.5% compared to the original model.

[0046] In domain-specific tasks such as mathematical reasoning, using calibration data from the matching domain for hook training can further improve task performance and has a better compensation effect compared to using a general dataset.

[0047] In summary, this embodiment effectively mitigates the performance loss caused by MoE inference acceleration by introducing the Layer-Depth Awareness Expert Number Adjustment (TLNLE) strategy and a lightweight Hook module, with minimal calibration data and parameter changes, and possesses good scalability and engineering feasibility.

[0048] In a preferred embodiment, this method is applied to in-hospital deployment scenarios of multimodal medical analysis models. For example, hospital radiology / pathology departments, under limited computing power (such as a single graphics card or a small inference server), need to quickly analyze patient examination data and generate structured or textual conclusions.

[0049] Taking "medical imaging + clinical information → automatic generation of radiological conclusions / report highlights" as an example: the system receives the patient's CT / MR / DR / ultrasound images (or pathological slides / endoscopic frames), and can optionally receive structured test indicators (such as blood routine, biochemistry) and medical record summaries; the model output includes, but is not limited to: description of key imaging features, possible diagnostic tendencies, grading suggestions, and interpretable evidence corresponding to the output conclusions (such as areas of interest / sign alignment segments).

[0050] In this embodiment, the multimodal medical analysis model includes a visual encoder, a cross-modal alignment / fusion module, and a decoding module employing MoE. This invention accelerates inference and preserves accuracy for the MoE substructure (the decoding module employing MoE) in the multimodal medical analysis model.

[0051] In the aforementioned multimodal medical scenarios, the computational load of the model's inference is mainly concentrated in the multi-expert computation of the MoE feedforward layer. In this embodiment, the MoE layer adopts a Top-2 expert activation strategy to ensure expressive power, but this will bring significant inference latency and memory / throughput pressure.

[0052] Therefore, this embodiment adopts the TLNLE strategy to determine the range of MoE layers that reduce the number of activated experts, and reduces the number of activated experts in the MoE layers within this range from 2 to 1, so as to reduce the overall number of expert calls and computational load.

[0053] Medical multimodal tasks not only require accurate text but also reliable alignment between "image evidence and textual conclusions." Reducing expert activation can easily lead to a shift in the distribution of cross-modal attention and fused representations, manifesting as: generalization of conclusion descriptions, weakened evidence alignment, or a shift in the region of interest.

[0054] This invention focuses on restoring the consistency between fused representation and evidence alignment through compensatory learning of the Hook module. Specifically, it selects the cross-modal fusion key layer and / or language decoding key layer, which are more sensitive to performance, from the MoE substructure as the insertion layer for the Hook module; the Hook module is inserted into this MoE layer to compensate for the intermediate representation shift caused by the reduction of Top-k, thereby maintaining the stability of the multimodal fusion representation.

[0055] Then, in this embodiment, a small-scale cross-modal calibration set is used to train the Hook module. The calibration samples include a small amount of paired data: multi-source aligned samples of "medical images / pathological slides / endoscopic frames + corresponding examination conclusions / radiological reports / pathological diagnostic points" and "images + structured test indicators + medical record summaries".

[0056] The training objective of the Hook module is to use the state space approximation / representation approximation approach, that is, after reducing expert activation, the intermediate representation of the Hook module should be as close as possible to the cross-modal fusion representation of the original Top-2 model in the corresponding layer, thereby maintaining the consistency of medical multimodal task output and the ability to align interpretable evidence while reducing computation.

[0057] Without altering the original visual encoder and cross-modal attention structure, this embodiment achieves the following: 1. Accelerated inference and reduced computational power: By reducing the number of MoE expert activations, the amount of feedforward computation and memory bandwidth consumption are reduced, thereby increasing throughput or reducing latency; 2. Preservation of accuracy and consistency: The Hook module reduces the performance loss caused by Top-k descent by compensating for the key layer representation shift; 3. Enhanced interpretable evidence alignment: By approximating the cross-modal fusion representation, the alignment capability between the output conclusion and image evidence (area of ​​interest / signs) is made more stable; 4. Pluggable and portable deployment: The Hook module has a small number of parameters and can be quickly adapted to different equipment / departmental distribution differences using a small amount of in-hospital calibration data, achieving lightweight migration and rapid deployment.

[0058] In summary, this method, without altering the original MoE architecture, controls and expands only the actual number of activated experts in each MoE layer and the insertion of Hook modules in designated layers, achieving inference acceleration technology with reduced experts, lightweight compensation, and rapid calibration with small data, to support low-latency deployment for clinical use.

[0059] This embodiment also provides an inference acceleration and performance compensation system for hybrid expert models, and the technical solution adopted is as follows: including: The module for adjusting the number of activated experts is used to obtain the model to be optimized. Under the premise that the performance loss of the model to be optimized is less than the performance loss threshold, several MoE layers of the model to be optimized are selected to reduce the number of activated experts.

[0060] The Hook module is used to select MoE layers that are sensitive to the number of experts and add Hook modules to them. The Hook module is located before the expert modules of the MoE layer. The Hook module adopts a residual structure with two layers of linear transformation and nonlinear activation.

[0061] The model training module is used to train the Hook module to obtain the optimized model.

[0062] The model to be optimized is a multimodal medical analysis model. The patient's CT / MR / DR / ultrasound images (or pathological slides / endoscopic frames), as well as structured test indicators (such as blood routine and biochemistry) and medical record summaries are input into the optimized model; after inference, the optimized model outputs descriptions of key imaging features, possible diagnostic tendencies, grading suggestions, and interpretable evidence corresponding to the output conclusions.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accelerating inference and compensating performance for hybrid expert models, characterized in that, include: S1: Obtain the model to be optimized; S2: Provided that the performance loss of the model to be optimized is less than the performance loss threshold, select several MoE layers of the model to be optimized to reduce the number of activated experts. S3: Select the MoE layer which is more sensitive to the number of experts and add a Hook module; the Hook module is located before the expert module of the MoE layer; the Hook module adopts a residual structure with two layers of linear transformation and nonlinear activation; S4: Train the Hook module to obtain an optimized model, and use the optimized model for inference.

2. The inference acceleration and performance compensation method for hybrid expert models as described in claim 1, characterized in that, The model to be optimized is a multimodal medical analysis model that has already been trained.

3. The inference acceleration and performance compensation method for hybrid expert models as described in claim 2, characterized in that, The model to be optimized includes multiple MoE layers, and each MoE layer contains multiple expert modules, which are feedforward neural networks.

4. The inference acceleration and performance compensation method for hybrid expert models as described in claim 1, characterized in that, In step S3, the process iterates from the last MoE layer to the first MoE layer; the number of activated experts is reduced in the traversed MoE layers, and the performance loss of the model to be optimized is evaluated; the first M MoE layers with performance loss not less than the performance degradation threshold are selected and added to the Hook module.

5. The inference acceleration and performance compensation method for hybrid expert models as described in claim 4, characterized in that, M is 1.

6. A method for inference acceleration and performance compensation for hybrid expert models as described in claim 1 or 2, characterized in that, The Hook module hides the input state. The processing method is as follows: in, This is the first trainable linear layer. The second trainable linear layer, It is a non-linear activation function. Scalar scaling factor This refers to the output characteristics of the Hook module.

7. The inference acceleration and performance compensation method for hybrid expert models as described in claim 6, characterized in that, In step S4, the model to be optimized is frozen, and only the Hook module and the scalar scaling factor are updated.

8. A method for inference acceleration and performance compensation for hybrid expert models as described in claim 1 or 2, characterized in that, In step S2, the expert activation strategy is to activate the top-k experts with the highest probability of activating the route output; The Lth layer to the last MoE layer of the model to be optimized is set to use kn experts, while the remaining MoE layers keep k experts. When L takes different values, the performance loss of the model to be optimized is tested respectively, and the L value with the largest performance loss less than the performance loss threshold and the largest reduction in the number of activated experts is obtained.

9. The inference acceleration and performance compensation method for hybrid expert models as described in claim 8, characterized in that, The number of activation experts in the MoE layer was reduced from 2 to 1.

10. A reasoning acceleration and performance compensation system for hybrid expert models, characterized in that, A method for accelerating inference and compensating performance for hybrid expert models as described in any one of claims 1 to 9, comprising: The expert number adjustment module is used to obtain the model to be optimized; provided that the performance loss of the model to be optimized is less than the performance loss threshold, select several MoE layers of the model to be optimized and reduce the number of activated experts. The Hook module is used to select MoE layers that are sensitive to the number of experts and add Hook modules to them; the Hook module is located before the expert modules of the MoE layer; the Hook module adopts a residual structure with two layers of linear transformation and nonlinear activation. The model training module is used to train the Hook module to obtain the optimized model.

Images