Task-specific augmented hybrid expert model and training method

Abstract

The application discloses a task-specific enhanced hybrid expert model and a training method, relates to the technical field of hybrid expert models, and provides general knowledge support and outputs base features first, is connected with the base model through a residual injection mode, learns and outputs task-specific features, then generates activation strength based on input Prompt global features, filters and aggregates expert outputs, then respectively fine-tunes task-specific knowledge of independent fine-tuning task experts and binary classification head recognition ability, and finally activates experts through a double-path judgment, superimposes and aggregates outputs through residual injection to obtain a final result. Through the general expert module based on the pre-training stage, the scheme increases the task expert directional enhancement paradigm in the fine-tuning stage, realizes the accurate improvement of specific task performance under the premise that the general ability of the model is not damaged during fine-tuning, and supports the rapid fusion of multiple task version fine-tuning experts.

Description

Technical Field

[0001] This invention relates to the field of hybrid expert model technology, specifically to a task-specific enhanced hybrid expert model and its training method. Background Technology

[0002] Hybrid expert models (MOEs) expand model capabilities through parallel processing of multiple experts and are a widely adopted architecture for large language models. However, existing MOEs have core pain points: MOEs select experts on a token-by-token basis, making it impossible to precisely pinpoint changes to a specific expert when fine-tuning for different tasks. Furthermore, the parameters in the pre-training and fine-tuning stages of large language models are tightly coupled, making it easy for fine-tuning to lead to the forgetting of general knowledge, and the model parameters fine-tuned for different tasks are difficult to integrate with each other.

[0003] Existing technologies, such as the invention patent application with publication number CN118821846A, disclose an implementation method and strategy for a hybrid expert network based on information segments. This hybrid expert network includes multiple candidate expert networks. The implementation method includes: determining whether any output of the hybrid expert network is a preset information segment end symbol; when any output is an information segment end symbol, using a pre-trained expert selection network, based on a first processing result obtained after performing a first processing on all outputs of the current information segment, selecting at least one from the multiple candidate expert networks as the current expert network; when any output is not the information segment end symbol, the current expert network remains unchanged; using any output as input, and activating the hybrid expert network of the current expert network, performing the next inference. Applying this application allows for more rational expert network selection, effectively improving the efficiency of the entire system.

[0004] As can be seen from the above solutions, existing fine-tuning methods for hybrid expert models can achieve performance enhancement on specific tasks, but the model's original knowledge and general capabilities are usually forgotten and weakened, and cannot coexist on the same model; different tokens of the same semantic unit are assigned to different experts, causing experts to be unable to capture sequence-level global semantic relationships; after fine-tuning for a specific task, parameter changes cannot be located to a specific expert; at the same time, the parameters fine-tuned for different tasks are difficult to integrate, and the model cannot maintain multiple specific capabilities at the same time. Summary of the Invention

[0005] To address the aforementioned technical shortcomings, the present invention aims to provide a task-specific enhanced hybrid expert model and training method.

[0006] To solve the above technical problems, the present invention adopts the following technical solution: The present invention provides a task-specific enhanced hybrid expert model and training method, including the following steps: S1, providing general knowledge support and outputting base features, using a frozen hybrid expert Transformer architecture, wherein the Attention layer, routing and original expert parameters of the base model are all fixed.

[0007] S2. Connect to the base model via residual injection, learn and output task-specific features, form a dynamically expandable structure, and include at least one task expert, with each task expert corresponding to an independent task binary classification head.

[0008] S3. Generate the activation intensity of each task expert based on the global features of the input Prompt, filter the set of activated task experts according to the activation intensity, and perform weighted aggregation on the output of the activated task experts.

[0009] S4. Independently fine-tune the task-specific knowledge of the training task experts and the task recognition ability of the task binary classification head.

[0010] S5. Calculate the global feature vector, activate the task expert through rule-triggered and model-triggered dual-path judgment, and superimpose the aggregated task expert output onto the base model output through residual injection to obtain the final inference result.

[0011] The beneficial effects of this invention are as follows: 1. This invention provides a task-specific enhanced hybrid expert model and training method. First, it provides general knowledge support and outputs base features. Then, it connects to the base model via residual injection to learn and output task-specific features. Next, it generates activation strengths based on the input Prompt global features, filters and aggregates expert outputs, and then fine-tunes the task-specific knowledge and binary classification head recognition capabilities of independently trained experts. Finally, it determines the activation experts through dual-path determination, and obtains the final result through residual injection superposition and aggregation. This scheme, based on a general expert module in the pre-training stage, adds a task-oriented expert enhancement paradigm in the fine-tuning stage, achieving precise improvement of specific task performance without destroying the model's general capabilities during fine-tuning, and supports rapid fusion of fine-tuned experts from multiple task versions.

[0012] 2. During the fine-tuning phase, the parameters of the general experts and routes are fixed, and only the newly added task experts are fine-tuned with small-scale incremental updates. This achieves decoupled storage of general knowledge and task knowledge; when reasoning for non-target tasks, only the general experts are enabled, completely avoiding the interference of task-specific parameters on general capabilities, and ensuring that the model maintains stable generalization performance in cross-task scenarios.

[0013] 3. In the fine-tuning phase, a task-expert-oriented activation mechanism is used to enable dedicated task experts to focus on learning knowledge specific to a single task, combined with the common knowledge support provided by general experts, to achieve knowledge complementarity between "general" and "dedicated".

[0014] 4. When adding a new task, you only need to add the corresponding task expert and its classification head through the plugin, without adjusting the parameters of the general expert or the existing task expert. This incremental learning capability enables the model to flexibly adapt to multi-task scenarios, reduce the development cycle and training cost of subsequent tasks, and is especially suitable for enterprise-level application scenarios that require continuous iteration to add new tasks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1 As shown, a task-specific enhanced hybrid expert model and training method are described, including the following steps: S1, providing general knowledge support and outputting base features, using a frozen hybrid expert Transformer architecture, wherein the Attention layer, routing and original expert parameters of the base model are all fixed.

[0019] In a specific embodiment, the specific process of S1 is as follows: Infrastructure: A frozen hybrid expert Transformer is adopted, and the original Attention, routing and experts are all frozen to ensure that the core capabilities do not degrade.

[0020] Base output formula: The output of the original hybrid expert at layer l is calculated as follows: ,in, This represents the hidden input state of this layer. For the first in the base model An original expert, Assign base routes to experts Gating weights, Used as an index to traverse the collection. Every original expert in it.

[0021] It should be noted that the hybrid expert Transformer refers to a model infrastructure that integrates the hybrid expert (MoE) architecture and the Transformer architecture. Its core feature is that multiple parallel expert modules process the input data separately, and the Transformer's Attention mechanism captures semantic relationships, thereby achieving model capacity expansion and efficient computation.

[0022] Parameter freezing: refers to the technical means of fixing the parameters of a specified module and not updating them during model training or inference. In this invention, the parameters of the core components of the base model are frozen in order to retain the general knowledge learned in the pre-training stage and avoid interference with the general capabilities caused by subsequent task fine-tuning, i.e., catastrophic forgetting.

[0023] Original Attention: The core component of the Transformer architecture, used to calculate the association weights of tokens at different positions in the input sequence, so as to focus on key information and capture semantic dependencies; Original Attention specifically refers to the pre-trained Attention layer in the base model, which is distinguished from the task-related components that may be added later, and its parameters are frozen to ensure that the general semantic capture capability does not degrade.

[0024] The expert selection result set for the base routing refers to the set of experts selected by the routing module from all original experts, based on the current input. The highest matching degree One original expert ( (This is a preset hyperparameter, representing the number of experts participating in parallel computation). Only experts within this set participate in the base output computation.

[0025] S2. Connect to the base model via residual injection, learn and output task-specific features, form a dynamically expandable structure, and include at least one task expert, with each task expert corresponding to an independent task binary classification head.

[0026] In a specific embodiment, the specific process of S2 is as follows: Using aggregated residual injection logic, new task experts are injected into the model in a residual manner. The final output is obtained by aggregating the outputs of multiple task experts, and the calculation formula is: ,in, This is a weighted aggregate output of multiple task experts, where λ is a fixed coefficient, much smaller than the original expert strength.

[0027] It should be noted that the aggregated residual injection logic refers to the combined feature fusion logic that first aggregates the outputs of multiple task experts in a weighted manner, and then injects the output of the base model in the form of residuals. The core is the dual design of aggregation and residuals: first, the specific features of multiple related task experts are integrated in a weighted manner, and then the general features of the base model are integrated in the form of residual superposition.

[0028] Residual injection: This involves injecting new features from the aggregated output of the task expert into the original output of the base model through direct superposition to form the final output. The core advantage is that it does not destroy the original feature expression, fully preserves the general features of the base model, and adds the specific features of the task expert only as supplementary increments.

[0029] Task Expert: This refers to a newly added independent sub-module in the Task Expert Pool module that focuses on learning knowledge of a specific task domain. Each Task Expert corresponds to an independent task binary classification head and participates in the calculation only when the task is activated. Its core function is to capture and output task-specific features.

[0030] Weighted aggregation output: refers to the fusion feature vector obtained by weighting the outputs of multiple activated task experts by assigning relative weights according to their activation strength. Weighting means that the higher the activation strength of a task expert, the higher the proportion of its output in the aggregation result. Aggregation means that it integrates the specific features of multiple related task experts to form a unified task-related feature output, ensuring that the output magnitude is stable when multiple experts work together.

[0031] Fixed coefficient (λ): A preset constant used to adjust the intensity of the aggregated output contribution of the task experts. Its value is much smaller than the output intensity of the original experts, such as 0.01-0.1. Its core function is to limit the influence of the task expert output on the base model output.

[0032] S3. Generate the activation intensity of each task expert based on the global features of the input Prompt, filter the set of activated task experts according to the activation intensity, and perform weighted aggregation on the output of the activated task experts.

[0033] In a specific embodiment, the process of generating the activation intensity of each task expert based on the global features of the input Prompt is as follows: a task classification mechanism based on the global features of the Prompt is used to accurately identify the task domain and intensity.

[0034] Each task expert Corresponding to an independent binary classification head .

[0035] Input features: Take the hidden layer output of the Prompt after passing through the last layer of the Transformer. Global pooling is performed to obtain the feature vector. : .

[0036] Strength calculation: Each classification head outputs the probability / strength of the input belonging to the corresponding task. : ,in For the Sigmoid function, , and For binary classification header parameters, As an index, representing the first Task expert ( ) and its bound first Two-classification heads ( ).

[0037] It should be noted that Prompt global features refer to the complete semantic features extracted from the input Prompt after processing by the Transformer architecture (rather than the local features of a single token), including global information such as the core intent of the Prompt and task domain attributes.

[0038] Binary classification head: refers to an independent computing unit bound to each task expert. Its core function is not the traditional binary classification of class determination, but rather based on the input global feature vector. The output is the matching probability / intensity between the input and the corresponding task.

[0039] Global pooling: A feature compression operation that transforms the sequential hidden layer features output from the last layer of a Transformer into a fixed-dimensional vector, i.e., a feature vector. This eliminates the impact of sequence length differences on subsequent calculations; common implementation methods include Mean Pooling.

[0040] The Sigmoid function maps the linear transformation result of the binary classifier to the interval [0,1].

[0041] Activation strength: refers to the feature vector calculated using the binary classifier head. The degree of matching with the corresponding task is used to determine the final output strength value, which ranges from [0,1]. The closer to 1, the higher the matching degree between the input and the task; the closer to 0, the lower the matching degree.

[0042] In a specific embodiment, the process of filtering the set of task experts based on activation intensity is as follows: Activation determination: If Exceeding the preset task expert activation threshold If so, it is considered that the input requires the introduction of a task expert. The activated expert group is , The set of experts that have been activated.

[0043] Global consistency: It is based on the semantic features of the full Prompt. Generated in one go, during the reasoning process. It remains unchanged across all layers of the Transformer, serving as a global static weight for the corresponding task expert.

[0044] It should be noted that the preset task expert activation threshold is a critical value used to determine whether a task expert is activated. It is set by professionals according to the judgment requirements, and no specific numerical limit is set here.

[0045] In a specific embodiment, the weighted aggregation process of the activated task expert outputs is as follows: Weighted average aggregation: If multiple task experts are activated, the aggregation is performed according to the strength of each expert. Calculate the relative weights and perform a weighted average aggregation of expert outputs to ensure that experts from multiple related tasks collaborate and that the output magnitude remains stable. The aggregation output calculation formula is as follows: .

[0046] It should be noted that weighted average aggregation is a set of task experts based on activation. The output of each expert ( ) and corresponding activation intensity Multiply the results, sum them, and then divide by the sum of the activation intensities of all experts in the set to obtain the final aggregated output.

[0047] Relative weights: refer to weights based on the activation set. Activation intensity of experts in each mission The calculated proportional weights, i.e., the individual expert's... Divide by all experts in the set The sum is used to quantify the contribution percentage of each expert in the aggregated output.

[0048] S4. Independently fine-tune the task-specific knowledge of the training task experts and the task recognition ability of the task binary classification head.

[0049] In a specific embodiment, the specific process of S4 is as follows: the fine-tuning process is divided into two independent parts: expert knowledge acquisition and task recognition capability construction, as follows: expert individual training: independently fine-tuning new experts on specific task data. At this stage, the base model is completely frozen, and only one task expert is configured. By capturing domain-specific knowledge for the task through supervised learning, the task expert output satisfies the following: .

[0050] Binary Classifier Head Training: Training Binary Classifier Head Parameters Using Mixed Task Data and Training input: Feature vector after the last pooling layer .

[0051] Training objective: To ensure that the binary classification head for the input task outputs high confidence, and the binary classification head for unrelated tasks outputs low confidence.

[0052] The loss function used for training is the task classification loss, specifically the binary cross-entropy loss for each classifier head.

[0053] It should be noted that high confidence refers to the output result of the binary classification head for the corresponding task of the input. A state close to 1 indicates that the binary classifier determines that the input is highly matched with the current task, and accurately identifies the corresponding task in the quantification training objective.

[0054] Low confidence: refers to the output of the binary classification head for tasks unrelated to the input. A state close to 0 indicates that the binary classifier determines that the input is not related to the current task, thus distinguishing irrelevant tasks in the quantification training objective.

[0055] Task classification loss: refers to the loss function used to optimize the task recognition capability of the binary classifier head. Its core function is to quantify the activation intensity of the binary classifier head's output. The difference between the input and the actual task attribution.

[0056] Binary cross-entropy loss: refers to the specific implementation of task classification loss. It is a classic loss function adapted for binary classification, i.e., whether the input belongs to a certain task, and quantifies the probabilistic output. The difference between ∈[0,1] and the true label.

[0057] S5. Calculate the global feature vector, activate the task expert through rule-triggered and model-triggered dual-path judgment, and superimpose the aggregated task expert output onto the base model output through residual injection to obtain the final inference result.

[0058] In a specific embodiment, the specific process of calculating the global feature vector is as follows: Expert pool configuration: According to the specific scenario requirements, the user selects several expert modules and their corresponding binary classification heads from the trained expert library through a plug-in configuration method to form a dedicated expert pool for the current task.

[0059] Forward calculation of base: Input The output of the last layer is recorded as it flows through the Transformer. And calculate the global feature vector. , This is a global pooling operation.

[0060] It should be noted that the plug-in configuration method refers to the operation mode of users building their own expert pool. There is no need to modify the core architecture of the model. Users only need to select the required expert modules and corresponding binary classification heads from the trained expert library, and quickly configure them like installing a plug-in to adapt to the task requirements of different scenarios.

[0061] Trained expert database: refers to the resource set that stores all task expert modules that have been fine-tuned and trained to have specific task capabilities and their corresponding binary classification heads.

[0062] Dedicated expert pool: refers to a targeted set of experts selected and combined from the trained expert database based on the specific needs of the current scenario. It only includes experts and binary classification heads that are relevant to the current task, which can reduce the computational consumption of irrelevant experts.

[0063] Forward computation: The core computational process of model inference, which refers to the triggering priority of the input data passing through the Attention layer and expert module for feature transformation and computation, and finally outputting intermediate features or the final result.

[0064] In a specific embodiment, the specific process of activating the task expert through dual-path determination of rule triggering and model triggering is as follows: the task trigger determination includes rule triggering and model triggering, wherein rule triggering has a higher priority than model triggering.

[0065] Rule Triggering: In a specific and defined task scenario, the system checks whether the Prompt matches a trigger word in the preset rule module. If a matching trigger word is detected, the system bypasses the model's judgment, forcibly selects the corresponding task expert, and sets the strength of that expert. .

[0066] Model Triggering: If the Prompt does not match any preset rule trigger words, the activation intensity of the corresponding expert is calculated in parallel by the binary classification heads of each task. And filter out Task expert group activating expert group .

[0067] It should be noted that trigger priority refers to the execution order rule of the two trigger paths. Rule triggering takes precedence over model triggering. If both the rule triggering condition and the model triggering condition are met simultaneously, only the rule triggering condition will be executed.

[0068] Forced selection: refers to the expert activation method when the rule is triggered, without considering the activation strength. The comparison results with the preset task expert activation threshold directly lock the corresponding task expert and include it in the activation set, while forcibly setting... =1.0.

[0069] Parallel computing: refers to the simultaneous and independent calculation of the corresponding expert activation intensity by each task's binary classification head when the model is triggered. The calculation method, unlike serial calculation, can improve the calculation efficiency of activation intensity and shorten the inference time.

[0070] In a specific embodiment, the process of superimposing the aggregated task expert output onto the base model output via residual injection is as follows: Expert calculation and aggregation: only for the selected or forcibly chosen expert set. Calculations were performed based on the activation intensity of each expert. Perform a weighted average, where the rule-triggered experts This yields the aggregated output from each layer. .

[0071] Residual injection: This will aggregate the output. Multiply by a fixed coefficient The output is superimposed onto the original base model, and the final output calculation formula is: .

[0072] It should be noted that the set of experts selected through filtering or forced selection refers to the final set of task experts determined after the dual-path triggering of the inference module, which consists of two parts: one is the set of experts that meet the requirements under model triggering. > The experts are either experts who are forcibly activated under the triggering of rules, or experts who are experts who are forcibly activated under the triggering of rules.

[0073] The examples described in this invention are not limited to the specific embodiments listed above. The examples are merely illustrative to facilitate understanding of the invention and do not constitute a limitation on the scope of protection of this invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of this invention should be included within the scope of protection.

[0074] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in this specification, they should all fall within the protection scope of the present invention.

Claims

1. A task-specific enhanced hybrid expert model and its training method, characterized in that, Includes the following steps: S1. Provide general knowledge support and output base features. The base model adopts a frozen hybrid expert Transformer architecture. The Attention layer, routing and original expert parameters of the base model are all fixed. S2. Connect to the base model through residual injection, learn and output task-specific features, and form a dynamically expandable structure containing at least one task expert, with each task expert corresponding to an independent task binary classification head. S3. Generate the activation intensity of each task expert based on the global features of the input Prompt, filter the set of activated task experts according to the activation intensity, and perform weighted aggregation on the output of the activated task experts. S4. Independently fine-tune the task-specific knowledge of the training task expert and the task recognition capability of the task binary classification head. S5. Calculate the global feature vector, activate the task expert through rule-triggered and model-triggered dual-path judgment, and superimpose the aggregated task expert output onto the base model output through residual injection to obtain the final inference result.

2. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of S1 is as follows: Infrastructure: A frozen hybrid expert Transformer is used, with the original Attention, routing, and experts all frozen to ensure that core capabilities do not degrade; Base output formula: The output of the original hybrid expert at layer l is calculated as follows: ,in, This represents the hidden input state of this layer. For the first in the base model An original expert, Assign base routes to experts Gating weights, Used as an index to traverse the collection. Every original expert in it.

3. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of S2 is as follows: Using an aggregated residual injection logic, new task experts are injected into the model as residuals. The final output is obtained by aggregating the outputs of multiple task experts, and the calculation formula is as follows: ,in, This is a weighted aggregate output of multiple task experts, where λ is a fixed coefficient, much smaller than the original expert strength.

4. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of generating the activation intensity of each task expert based on the global features of the input Prompt is as follows: A task classification mechanism based on global Prompt features is used to accurately identify the task domain and intensity. Each task expert Corresponding to an independent binary classification head ; Input features: Take the hidden layer output of the Prompt after passing through the last layer of the Transformer. Global pooling is performed to obtain the feature vector. : ; Strength calculation: Each classification head outputs the probability / strength of the input belonging to the corresponding task. : ,in For the Sigmoid function, , and For binary classification header parameters, As an index, representing the first Task expert ( ) and its bound first Two-classification heads ( ).

5. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of selecting the set of task experts based on activation intensity is as follows: Activation determination: If Exceeding the preset task expert activation threshold If so, it is considered that the input requires the introduction of a task expert. The activated expert group is , For the activated set of experts; Global consistency: It is based on the semantic features of the full Prompt. Generated in one go, during the reasoning process. It remains unchanged across all layers of the Transformer, serving as a global static weight for the corresponding task expert.

6. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of weighted aggregation of the activated task expert output is as follows: Weighted average aggregation: If multiple task experts are activated, the aggregation is based on the strength of each expert. Calculate the relative weights and perform a weighted average aggregation of expert outputs to ensure that experts from multiple related tasks collaborate and that the output magnitude remains stable. The aggregation output calculation formula is as follows: .

7. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of S4 is as follows: The fine-tuning process is divided into two independent parts: expert knowledge acquisition and task recognition capability building, as detailed below: Individual expert training: independently fine-tuning new experts on specific task data At this stage, the base model is completely frozen, and only one task expert is configured. By capturing domain-specific knowledge for the task through supervised learning, the task expert output satisfies the following: ; Binary Classifier Head Training: Training Binary Classifier Head Parameters Using Mixed Task Data and : Training input: Feature vector after the last pooling layer ; Training objective: To ensure that the binary classification head for the corresponding task outputs high confidence, and the binary classification head for unrelated tasks outputs low confidence; The loss function used for training is the task classification loss, specifically the binary cross-entropy loss for each classifier head.

8. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process for calculating the global feature vector is as follows: Expert pool configuration: Users can select several expert modules and their corresponding binary classification heads from the trained expert library according to specific scenario requirements through a plug-in configuration method to build a dedicated expert pool for the current task. Forward calculation of base: Input The output of the last layer is recorded as it flows through the Transformer. And calculate the global feature vector. , This is a global pooling operation.

9. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of activating the task expert through a dual-path decision-making process involving rule triggering and model triggering is as follows: Task triggering determination includes rule triggering and model triggering, with rule triggering having higher priority than model triggering. Rule Triggering: In a specific and defined task scenario, the system checks whether the Prompt matches a trigger word in the preset rule module. If a matching trigger word is detected, the system bypasses the model's judgment, forcibly selects the corresponding task expert, and sets the strength of that expert. ; Model Triggering: If the Prompt does not match any preset rule trigger words, the activation intensity of the corresponding expert is calculated in parallel by the binary classification heads of each task. And filter out Task expert group activating expert group .

10. The task-specific enhanced hybrid expert model and training method according to claim 1, characterized in that, The specific process of superimposing the aggregated task expert output onto the base model output via residual injection is as follows: Expert calculation and aggregation: Only for filtered or forced selections of experts. Calculations were performed based on the activation intensity of each expert. Perform a weighted average, where the rule-triggered experts This yields the aggregated output from each layer. ; Residual injection: This will aggregate the output. Multiply by a fixed coefficient The output is superimposed onto the original base model, and the final output calculation formula is: .

Images