A model initialization method based on cross-task shared expert inheritance

Abstract

The application provides a model initialization method based on cross-task shared expert inheritance, constructs a multi-task mixed expert ancestor model and trains; each task data is input into the ancestor model, and the selection probability of each expert is counted according to layers; the importance score is constructed based on the cross-task average selection probability and the entropy of the expert, and the shared expert is selected according to layers to form a learning gene; each layer of the shared expert is used as a basic expert, a derived expert set is generated through linear expansion to initialize the mixed expert module of the corresponding layer of the offspring model; in the training or fine-tuning of the offspring model, Top-K sparse routing is used to activate the expert and output; when the multi-task training is performed, the task-expert mutual information loss and the task loss are introduced to form a total loss and update the model parameters, and the convergence speed and the generalization performance of the application in unknown downstream tasks can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a model initialization method based on cross-task expert inheritance. Background Technology

[0002] With the widespread application of deep learning, especially Vision Transformer (ViT), in visual tasks, the industry often adopts a paradigm of large-scale pre-training followed by downstream fine-tuning. This involves pre-training on large datasets, such as classification datasets, to obtain parameter initializations, and then fine-tuning on the target task. However, this paradigm is prone to the following problems: 1) Task bias in pre-training: Pre-training often creates a strong bias for specific tasks, such as classification, making the parameter space more inclined towards the optimal subspace of that task. This can lead to situations where the downstream task differs significantly from the pre-training task, such as dense prediction (semantic segmentation, depth estimation, edge detection), where the pre-trained representation may actually inhibit adaptation and the learning of new features; this constraint becomes more pronounced as the task gap widens. 2) Insufficient cross-task generalization: Different types of visual tasks share low-level / mid-level visual patterns (edges, textures, geometry) and exhibit strong task specificity; traditional initialization methods struggle to explicitly extract "cross-task shared knowledge." 3) While Mixture-of-Experts (MoE) is efficient, its knowledge transfer mechanism is insufficient: MoE improves capacity / efficiency by sparsely activating experts, but existing works mostly focus on computational efficiency or single-task performance, lacking a systematic method to extract cross-task shared experts from multi-task MoE and use them for downstream initialization.

[0003] Therefore, a new initialization / inheritance framework is urgently needed: one that can extract "cross-task shared" knowledge carriers from multi-task MoE ancestor models and use them in a parametrically efficient manner for descendant models of different sizes, thereby achieving faster convergence and better generalization on the target task (which can be non-overlapping with the ancestor training task). Summary of the Invention

[0004] Technical Problem: Existing initialization strategies based on a single pre-training task (such as classification) are prone to overfitting bias and limiting downstream adaptation when faced with a large gap between pre-training and fine-tuning tasks. Simultaneously, there is a lack of a mechanism to automatically identify cross-task shared experts from multi-task MoEs and "inherit / amplify" them for use in the initialization of descendant models, making it difficult to balance cross-task generalization and training efficiency. This invention proposes an efficient initialization method based on cross-task shared expert inheritance (which can be called the "initialization method based on cross-task expert inheritance" or the "ICE initialization method"). The overall idea is as follows: Construct and train a multi-task MoE ancestor model containing multi-task routing. In the ancestor model, the selection probability distribution of each task for each expert is statistically analyzed layer by layer. An importance score is calculated for each expert by combining the "average activation probability" and "cross-task uniformity (entropy)," and several "cross-task shared experts" with the highest scores are selected at each layer to form transferable "learning genes / knowledge genes." The selected shared expert parameters are linearly amplified / combined to generate several "derived experts," which are used to initialize the expert parameters of the corresponding MoE layers in descendant models, achieving efficient parameter inheritance. During the training / fine-tuning of the offspring model, a Top-K sparse routing mechanism is used for expert activation; in multi-task training, a "task-expert mutual information" regularization term is introduced to enhance task specificity and expert differentiation, thereby improving multi-task performance while ensuring the inheritance of shared knowledge.

[0005] Technical Solution: To achieve the above objectives, the technical solution adopted by this invention is: a model initialization method based on cross-task expert inheritance, comprising the following steps:

[0006] S1, Construct or obtain a multi-task MoE ancestor model. Obtain an ancestor model that has been trained on T tasks. Its backbone is the Visual Transformer (ViT), and it replaces the MoE module in at least one MLP sublayer. Each block of the ViT includes a Self-Attention Module (MSA) and an MLP module. The MSA output can be represented as:

[0007] (1),

[0008] in For hidden dimensions; MLP can be represented as:

[0009] (2),

[0010] in , , .

[0011] S2, determine the task set and statistical data set for the ancestor model. Let the ancestor model training task set be... To analyze routing behavior, separate the data for each task. Prepare sample set And standardize preprocessing and input formats.

[0012] S3 performs forward inference on the multi-task data, recording the selection probabilities of each expert at each level. The task data is then input into the ancestor model. For each layer (or each MoE block), record the probability / frequency of the routing network's choice of each expert. Let the... Layer MoE contains An expert, record the task. Lower Expert The probability of selection is:

[0013] (3),

[0014] And define experts Selection probability distribution across all tasks:

[0015] (4),

[0016] S4 calculates the expert's average activation rate across tasks. For the first... Layer experts Calculate its average selection probability across tasks:

[0017] (5),

[0018] in This indicates the overall activity level of the expert; a higher value indicates that the expert is used more frequently in multitasking.

[0019] S5 normalizes the task selection probabilities to obtain an efficient distribution. To measure the expert's balance across tasks, [the following is done]: Normalization:

[0020] (6),

[0021] S6 is the cross-task balance entropy metric for computational experts. Entropy:

[0022] (7),

[0023] in The larger the entropy, the more evenly the expert selects across different tasks, which is more in line with the characteristics of a shared expert; the smaller the entropy, the more task-specific the expert is.

[0024] S7, Construct the expert comprehensive importance scoring function. To avoid selecting experts who are only active on a few tasks, and also to avoid selecting experts who are almost inactive, define the comprehensive score:

[0025] (8),

[0026] in This is the balance coefficient.

[0027] S8, selecting cross-task sharing experts by layer to construct learning genes. For each layer of LLL, according to... Sort by high to low, select the top Each expert serves as a shared set of experts for this layer (e.g., ...). The shared expert parameters of all layers are aggregated to obtain the learning gene: , of which Layer learning genes are: . For the two selected shared experts; Corresponding to MoE-MLP and The parameter set.

[0028] S9. Construct the descendant model structure and determine its MoE expert number. Construct the descendant model to be initialized. Its network depth can differ from that of the ancestor model (e.g., 4 / 6 / 8 layers, etc.), but it contains MoE modules in the corresponding layers; let the descendant model be the... Layer MoE requires E experts, and the expert parameters are denoted as E. .

[0029] S10 performs linear expansion based on the learned genes to generate expert parameters for the progeny model. For each layer... Using two basic shared experts ( Linear generation of EEE derived experts. Introducing a learnable vector of derived coefficients:

[0030] (9),

[0031] And generate the first one as follows One derivative expert:

[0032] (10)

[0033] This yields a complete set of expert parameters for each layer of the MoE in the descendant model.

[0034] S11, initialize the descendant model with the extended derived expert parameters. (The parameters obtained in step S10 are then used for this step.) Write to the descendant model This is used as an initialization parameter in the corresponding layer MoE module; and... These parameters can be used as trainable parameters in subsequent optimization.

[0035] S12, Construct the routing / gating network for the descendant model and set the sparse activation strategy. Configure the routing network for the descendant model. In a multitasking scenario, it may include Individual task gating network For input Gated output The expert selection probability is obtained through Softmax, and then Top-K sparse selection is applied: ,in To activate the number of experts.

[0036] S13 defines the forward computation form of the MoE module. The MoE output in the descendant model can be represented as... ,in Indicates the first The expert's forward output includes / Corresponding parameters .

[0037] S14, Define the task loss function (single-task or multi-task). If it is a supervised learning task, given the task... The loss can be written as: ,in This can be cross-entropy, pixel-level loss, or regression loss, etc. In multi-task scenarios, a loss set is formed. .

[0038] S15 introduces task-expert mutual information regularization to enhance expert task specificity. To avoid convergence among derived experts and improve expert specialization, mutual information loss (the correlation term between task and expert) is introduced:

[0039] (11),

[0040] in For a certain level of expert group, For a set of tasks.

[0041] S16, Construct the overall optimization objective and jointly train the descendant models. The total loss is obtained by weighting and combining the multi-task loss with the mutual information term:

[0042] (12)

[0043] in For weight hyperparameters.

[0044] S17, Update the parameters of the descendant model based on the total loss and output the trained model. Gradient descent is used to update all trainable parameters of the descendant model, including: routing network parameters and derivation coefficients. And, if necessary, expert parameters, etc. ,in The learning rate is used to output the descendant model after training. Used for target task reasoning or further fine-tuning.

[0045] Compared to existing technologies, the beneficial effects of this invention are as follows: Through the above technical solution, the method of this invention, based on the inheritance mechanism of cross-task shared experts in a multi-task hybrid expert model, introduces more generalizable shared knowledge during the model initialization stage, enabling the descendant model to obtain better feature representation capabilities in different task scenarios. Compared with models that only use random initialization or knowledge distillation alone, under the same training data and conditions, the descendant model constructed using the method of this invention exhibits higher task performance in multiple tasks; simultaneously, it maintains more stable convergence characteristics during training, fully demonstrating the effectiveness of cross-task shared expert inheritance in improving the overall performance of the model. Attached Figure Description

[0046] Figure 1 This is a framework diagram of the method of the present invention;

[0047] Figure 2 This invention is compared with other existing learning gene methods on different tasks. Detailed Implementation

[0048] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] Example 1:

[0050] A model initialization method based on cross-task shared expert inheritance is illustrated in the following diagram: Figure 1 As shown. In this embodiment, a multi-task hybrid expert ancestor model trained on multiple visual tasks is first obtained. The ancestor model contains multiple hybrid expert modules, and different experts form task-sharing experts and task-specific experts during the multi-task training process. By performing cross-task expert activation statistics on each hybrid expert module in the ancestor model, experts that are frequently and uniformly activated in different tasks are selected as cross-task shared experts, and the parameters of the shared experts are extracted as learning genes.

[0051] Subsequently, a descendant model structure with a smaller parameter size than the ancestor model is constructed. The descendant model maintains the same network hierarchy as the ancestor model, with hybrid expert modules placed at corresponding positions. Multiple derived expert parameters are generated by linearly expanding the learning genes, used to initialize the expert parameters of each hybrid expert module in the descendant model, thus enabling the descendant model to inherit the cross-task general knowledge from the ancestor model during the initial training phase. In the downstream task training phase, the initialized descendant model is applied to the target task dataset, which is different in type from the task used in the ancestor model training phase and has no overlap with the dataset. During training, experts participating in the computation are dynamically selected for different input samples through a routing network, and a sparse activation strategy is used to limit the number of experts participating in each forward computation. In multi-task training scenarios, mutual information constraints between tasks and experts are introduced into the loss function to enhance the correlation between different tasks and experts, promoting expert division of labor. Through the above methods, the final descendant model is constructed and trained, enabling it to learn target task-related features based on inheriting cross-task shared expert knowledge, thereby improving the model's convergence speed and generalization performance on unseen tasks. Specific steps include the following:

[0052] S1, Construct or obtain a multi-task MoE ancestor model. Obtain an ancestor model that has been trained on T tasks. Its backbone is the Visual Transformer (ViT), and it replaces the MoE module in at least one MLP sublayer. Each block of the ViT includes a Self-Attention Module (MSA) and an MLP module. The MSA output can be represented as:

[0053] (1),

[0054] in For hidden dimensions; MLP can be represented as:

[0055] (2),

[0056] in , , .

[0057] S2, determine the task set and statistical data set for the ancestor model. Let the ancestor model training task set be... To statistically analyze routing behavior, for each task... Prepare sample set And standardize preprocessing and input formats.

[0058] S3 performs forward inference on the multi-task data, recording the selection probabilities of each expert at each level. The task data is then input into the ancestor model. For each layer (or each MoE block), record the probability / frequency of the routing network's choice of each expert. Let the... Layer MoE contains An expert, record the task. Lower Expert The probability of selection is:

[0059] (3),

[0060] And define experts Selection probability distribution across all tasks:

[0061] (4),

[0062] S4 calculates the expert's average activation rate across tasks. For the first... Layer experts Calculate its average selection probability across tasks:

[0063] (5),

[0064] in This indicates the overall activity level of the expert; a higher value indicates that the expert is used more frequently in multitasking.

[0065] S5 normalizes the task selection probabilities to obtain an efficient distribution. To measure the expert's balance across tasks, [the following is done]: Normalization:

[0066] (6),

[0067] S6 is the cross-task balance entropy metric for computational experts. Entropy:

[0068] (7),

[0069] in The larger the entropy, the more evenly the expert selects across different tasks, which is more in line with the characteristics of a shared expert; the smaller the entropy, the more task-specific the expert is.

[0070] S7, Construct the expert comprehensive importance scoring function. To avoid selecting experts who are only active on a few tasks, and also to avoid selecting experts who are almost inactive, define the comprehensive score:

[0071] (8),

[0072] in This is the balance coefficient.

[0073] S8, selecting cross-task sharing experts by layer to construct learning genes. For each layer of LLL, according to... Sort by high to low, select the top Each expert serves as a shared set of experts for this layer (e.g., ...). The shared expert parameters of all layers are aggregated to obtain the learning gene: , of which Layer learning genes are: . For the two selected shared experts; Corresponding to MoE-MLP and The parameter set.

[0074] S9. Construct the descendant model structure and determine its MoE expert number. Construct the descendant model to be initialized. Its network depth may differ from that of the ancestor model (e.g., 4 / 6 / 8 layers, etc.), but it contains MoE modules in the corresponding layers; let the descendant model be the... Layer MoE requires E experts, and the expert parameters are denoted as E. .

[0075] S10 performs linear expansion based on the learned genes, generating expert parameters for the progeny model. For each layer... Using two basic shared experts ( Linear generation of EEE derived experts. Introducing a learnable vector of derived coefficients:

[0076] .

[0077] And generate the first one as follows One derivative expert:

[0078] (9),

[0079] This yields a complete set of expert parameters for each layer of the MoE in the descendant model.

[0080] S11, initialize the descendant model with the extended derived expert parameters. (The parameters obtained in step S10 are then used for this step.) Write to the descendant model This is used as an initialization parameter in the corresponding layer MoE module; and... These parameters can be used as trainable parameters in subsequent optimization.

[0081] S12, Construct the routing / gating network for the descendant model and set the sparse activation strategy. Configure the routing network for the descendant model. In a multitasking scenario, it may include Individual task gating network For input Gated output The expert selection probability is obtained through Softmax, and then Top-K sparse selection is applied: ,in To activate the number of experts.

[0082] S13 defines the forward computation form of the MoE module. The MoE output in the descendant model can be represented as... ,in Indicates the first The expert's forward output includes / Corresponding parameters .

[0083] S14, Define the task loss function (single-task or multi-task). If it is a supervised learning task, given the task... The loss can be written as: ,in This can be cross-entropy, pixel-level loss, or regression loss, etc. In multi-task scenarios, a loss set is formed. .

[0084] S15 introduces task-expert mutual information regularization to enhance expert task specificity. To avoid convergence among derived experts and improve expert specialization, mutual information loss (the correlation term between task and expert) is introduced:

[0085] (10)

[0086] in For a certain level of expert group, For a set of tasks.

[0087] S16, Construct the overall optimization objective and jointly train the descendant models. The total loss is obtained by weighting and combining the multi-task loss with the mutual information term:

[0088] (11),

[0089] in For weight hyperparameters.

[0090] S17, Update the parameters of the descendant model based on the total loss and output the trained model. Gradient descent is used to update all trainable parameters of the descendant model, including: routing network parameters and derivation coefficients. And, if necessary, expert parameters, etc. ,in The learning rate is used to output the descendant model after training. Used for target task reasoning or further fine-tuning.

[0091] Test example:

[0092] In comparisons of the performance of the proposed method with random initialization and random initialization plus knowledge distillation on different tasks in ViT-small and ViT-tiny models at different layers, the model using the proposed method achieved optimal or significantly better performance across multiple task metrics. Specific results are shown in Table 1. Compared to random initialization methods using only a hybrid expert structure and methods incorporating knowledge distillation, the proposed method achieved stable improvements in mIoU metrics for tasks such as semantic segmentation, human body part segmentation, saliency detection, and edge detection, while also achieving lower error values ​​in the normal estimation task. This indicates that initialization through cross-task shared expert inheritance can more effectively improve the overall performance and representational capabilities of the model in multi-task scenarios.

[0093] Table 1 shows the multi-task performance comparison between the method of the present invention and the random initialization method on the PASCAL-Context dataset. The models used for comparison are the visual models ViT-small and ViT-tiny with sizes of 4, 6, and 8 layers, respectively.

[0094] Table 1

[0095]

[0096] Table 2 shows the performance comparison between the proposed method and random initialization and random initialization + knowledge distillation on different tasks in ViT-small and ViT-tiny models with different layers. Table 2 further verifies the effectiveness of the ICE method under different task combinations. In multi-task settings that simultaneously include semantic segmentation, depth estimation, and normal estimation, ICE significantly outperforms random initialization and traditional knowledge distillation methods across all model sizes and layer configurations. Especially with deeper networks, the ICE method achieves a greater improvement in the mIoU metric for segmentation tasks, while significantly reducing error values ​​in depth estimation and normal estimation tasks, demonstrating superior cross-task generalization ability and training stability. This indicates that the cross-task shared expert inheritance mechanism introduced by ICE can continue to exert its advantages in complex multi-task learning scenarios.

[0097] Table 2 shows the multi-task performance comparison between the method of the present invention and the random initialization method on the NYUD-v2 dataset. The models used for comparison are the visual models ViT-small and ViT-tiny with sizes of 4, 6, and 8 layers, respectively.

[0098] Table 2

[0099]

[0100] Figure 2 This paper presents a visual comparison of the performance of the method of this invention with other learning gene initialization methods on different downstream tasks, including edge detection, surface normal estimation, and saliency detection. The results in the figure show that the model initialized using the method of this invention can generate clearer, more structurally consistent predictions that conform to the semantics of the tasks, and its overall performance is significantly better than other learning gene methods. This is mainly due to the inheritance mechanism based on cross-task shared experts introduced in the model initialization stage of this invention, which can effectively extract and retain common knowledge between different tasks, thereby avoiding bias of the learning gene towards a single task and making the model more conducive to generalization to different downstream tasks.

[0101] In summary, this paper proposes a model initialization method based on cross-task shared expert inheritance to improve the overall performance of descendant models in multi-task and downstream tasks. By automatically identifying and inheriting shared experts across tasks from multi-task hybrid expert ancestor models, descendant models possess more generalizable feature representation capabilities from the initialization stage, effectively overcoming the limitations of traditional random initialization or single knowledge distillation methods in model performance. Addressing the significant differences between different tasks, this invention focuses on extracting and preserving common knowledge between tasks, avoiding bias in the initialization process towards a single task, and thus facilitating model generalization to different downstream tasks. Experimental results under various model sizes and multi-task settings demonstrate that, compared with existing initialization and distillation methods, this invention consistently achieves superior task performance, fully validating the effectiveness and versatility of the proposed method.

[0102] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A model initialization method based on cross-task shared expert inheritance, characterized in that, include: Construct and train a multi-task hybrid expert ancestor model; Input the data for each task into the ancestor model, and calculate the probability of each expert being selected by each layer. Importance scores are constructed based on the average selection probability and entropy of experts across tasks, and shared experts are selected by layer to form a learning gene; Each layer of shared experts is used as the base experts, and a derived expert set is generated through linear expansion to initialize the hybrid expert module of the corresponding layer of the descendant model. In the training or fine-tuning of the descendant model, Top-K sparse routing is used to activate the experts and output them. During multi-task training, task-expert mutual information loss and task loss are introduced to form a weighted total loss and update the model parameters.

2. The method of claim 1, wherein, Includes the following steps: S1, constructing or obtaining a multi-task MoE ancestor model, obtaining an ancestor model that has been trained on T tasks , the backbone of which is a visual Transformer (ViT) and is replaced by a MoE module in at least one MLP sublayer, each block of the ViT comprising a self-attention module (MSA) and an MLP module, the output of the MSA being represented as: （1）， wherein is the hidden dimension, are the query, key and value representations in the self-attention layer, respectively; and the MLP is represented as: （2）， in It is input. , , It is an activation function. It is a bias parameter. ; S2, determine the task set and statistical data set of the ancestor model, let the ancestor model training task set be... To statistically analyze routing behavior, separate the data for each task. Prepare sample set And standardize preprocessing and input formats; S3 performs forward inference on the multi-task data, records the selection probabilities of each expert at each layer, and inputs the task data into the ancestor model. For each layer or each MoE block, record the probability / frequency of the routing network's choice for each expert, and let the... Layer MoE contains An expert, record the task. Lower Expert The probability of selection is: （3）， And define experts Selection probability distribution across all tasks: （4）， S4, calculate the expert's average activation rate across tasks, for the first... Layer experts Calculate its average selection probability across tasks: （5）， in This indicates the overall activity level of the expert; a higher value indicates that the expert is used more frequently in multitasking. S5, normalize the task selection probabilities to obtain the efficient distribution. To measure the expert's balance across tasks, [the following is performed]. Normalization: （6）， S6, the cross-task balance entropy index for computational experts, computational expert Entropy: （7）， in The larger the entropy, the more evenly the expert selects across different tasks; the smaller the entropy, the more task-specific the expert is. S7, Construct the expert comprehensive importance scoring function and define the comprehensive score: （8）， in This is the balance coefficient; S8, selects cross-task sharing experts by layer, constructs learning genes, and for each layer lll, according to Sort by high to low, select the top Each expert is considered as a shared expert set for that layer. The shared expert parameters of all layers are then combined to obtain the learning gene. , of which Layer learning genes are: , For the two selected shared experts; Corresponding to MoE-MLP and The set of parameters; S9, Construct the descendant model structure and determine its MoE expert number, and construct the descendant model to be initialized. Its network depth may differ from that of the ancestor model, but it contains MoE modules in the corresponding layers; let the descendant model be the... Layer MoE requires E experts, and the expert parameters are denoted as E. ; S10 performs linear expansion based on the learned genes, generating expert parameters for the progeny model at each layer. Using two basic shared experts ( Linearly generate EEE derived experts, introducing a learnable vector of derived coefficients: （9）， And generate the first one as follows One derivative expert: （10）， This yields a complete set of expert parameters for each layer of the MoE in the descendant model; S11, initialize the descendant model with the extended derived expert parameters, and use the parameters obtained in step S10. Write to the descendant model This is used as an initialization parameter in the corresponding layer MoE module; and... As trainable parameters, they can be used in subsequent optimizations; S12, Construct the routing / gating network for the descendant model and set up a sparse activation strategy to configure the routing network for the descendant model. In a multitasking scenario, it may include Individual task gating network , for input Gated output The expert selection probability is obtained through Softmax, and then Top-K sparse selection is applied: ,in To activate the number of experts; S13 defines the forward computation form of the MoE module; the MoE output in the descendant model can be represented as follows: ,in Indicates the first The expert's forward output includes / Corresponding parameters ; S14, Define the task loss function. If it is a supervised learning task, given the task... The loss can be written as: In multi-task scenarios, a loss set is formed. ; S15 introduces task-expert mutual information regularization to enhance expert task specificity, and introduces mutual information loss: （11）， in For a certain level of expert group, For a set of tasks; S16, Construct the overall optimization objective and jointly train the descendant model, then weight and combine the multi-task loss with the mutual information term to obtain the total loss: （12）， in These are weight hyperparameters; S17 updates the parameters of the descendant model based on the total loss and outputs the trained model. Gradient descent is used to update all trainable parameters of the descendant model, including: routing network parameters and derivation coefficients. And, if necessary, expert parameters, etc. ,in The learning rate; Output the descendant model after training Used for target task reasoning or further fine-tuning.

3. The method as described in claim 2, characterized in that, In step S1, the ancestor model is trained using a multi-task hybrid expert model structure. During the training process, data samples from multiple visual tasks are input into the ancestor model. Different tasks correspond to independent or distinguishable routing strategies, enabling the ancestor model to form task-sharing experts and task-specific experts during multi-task training. In step S2, for each hybrid expert module in the ancestor model, the probability of each expert being selected by the routing network under different tasks is statistically calculated layer by layer according to the network hierarchy. The probability is obtained by the routing network after Softmax normalization.

4. The method as described in claim 2, characterized in that, In step S3, the expert's average activation probability across tasks is calculated as follows: the sum of the expert's activation probabilities across all tasks is divided by the number of tasks to characterize the expert's overall activity level in a multi-task scenario. In step S4, the activation probabilities of experts under different tasks are normalized, and the information entropy is calculated based on the normalized activation probability distribution as a measure of the balance of experts across tasks.

5. The method as described in claim 2, characterized in that, In step S5, the expert's overall importance score is composed of a weighted sum of the average activation probability and information entropy, where the weighting coefficient is used to adjust the relative weight of expert activity and cross-task balance in the score.

6. The method as described in claim 2, characterized in that, In step S6, in each hybrid expert module of the ancestor model, at least two experts are selected as cross-task shared experts, sorted from high to low according to their comprehensive importance scores, and their parameters are inherited as learning genes.

7. The method as described in claim 2, characterized in that, In step S7, multiple derived expert parameters are generated by linear combination using the parameters of the selected cross-task shared expert. Learnable expansion coefficients are introduced in the linear combination so that the derived experts have differences while inheriting shared knowledge.

8. The method as described in claim 2, characterized in that, In step S8, the hybrid expert module of the descendant model adopts the Top-K sparse routing strategy, which activates only the K experts with the highest routing probability to participate in the forward calculation, and resets the routing weight of the remaining experts to zero.

9. The method as described in claim 2, characterized in that, In step S9, in a multi-task training scenario, a mutual information loss term between tasks and experts is introduced to enhance the correlation between different tasks and experts and promote expert differentiation.

10. The method as described in claim 2, characterized in that: In step S10, the task loss function and the mutual information loss function are weighted and combined to form the overall loss function of the descendant model, and the parameters of the descendant model are optimized and updated based on the overall loss function.

Images