MoE expert model construction method and system based on domain task representation enhancement

Abstract

The invention discloses an MoE expert model construction method and system based on domain task representation enhancement. The method comprises the following steps: collecting multi-domain labeled data for training; extracting domain vectors through comparative learning to represent semantic features of different domains; introducing the vector into a hybrid expert model, fusing the vector with the output of the activated expert network, and carrying out secondary training on the model; converting the fusion features through a feature mapping layer, and inputting the converted fusion features into a large model subject network for calculation; and finally, training a complete model by using domain data to realize prediction of new data. According to the method, the task semantics and the MoE are combined through the domain vectors, so that adaptive expert selection and feature enhancement are realized, and the task processing precision of a large-scale model is improved.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a method and system for constructing a MoE expert model based on domain task representation enhancement. Background Technology

[0002] In recent years, artificial intelligence technology has developed rapidly, with methods represented by deep learning achieving significant breakthroughs in several key areas such as image recognition, natural language processing, and speech recognition. Large models (such as GPT and BERT), trained with massive parameters and large-scale data, have demonstrated powerful representation learning and generalization capabilities, driving the continuous improvement of industrial intelligence. Meanwhile, to address the increasing computational resource demands brought about by the expansion of model size, researchers are committed to exploring more efficient model architectures and training paradigms. Among these, Mixture-of-Experts (MoE) models have attracted widespread attention due to their scalability and flexibility, and have become an important technical approach in large-scale pre-training and multi-task learning.

[0003] In existing technologies, hybrid expert models typically employ a set of structurally identical or similar sub-networks (i.e., "experts") combined with gating mechanisms to achieve dynamic routing. Specifically, the gating network generates a weight distribution based on the input samples, selects to activate a subset of experts for processing, while the remaining experts remain dormant. This approach significantly reduces the number of parameters in forward computation while maintaining the model's expressive power. Typical implementations, such as Switch Transformer and GShard, are based on the sparse activation principle and achieve efficient allocation of different inputs through differentiable routing strategies (such as Softmax gating or Top-K selection), thus achieving excellent performance in tasks such as language modeling and machine translation.

[0004] However, while existing MoE methods perform well in single domains or similar tasks, their gating mechanisms often rely solely on shallow features of the input samples for routing decisions, lacking explicit modeling and utilization of task semantic information. This limitation makes it difficult for the model to accurately identify semantic differences and intrinsic relationships between different tasks when facing cross-domain or multi-task scenarios, resulting in problems such as unbalanced expert utilization and decreased generalization ability, thus limiting its applicability and scalability in real-world complex environments. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a domain task representation enhancement-based MoE expert model construction method that effectively combines task representation with MoE architecture to achieve efficient task adaptive expert routing and representation enhancement in large-scale pre-trained models; on the other hand, it provides a domain task representation enhancement-based MoE expert model construction system.

[0006] Technical solution: The MoE expert model construction method of the present invention includes the following steps:

[0007] S1. Collect several pre-defined data with domain labels from different domains for training a pre-trained language model;

[0008] S2. Based on the domain-labeled data collected in step S1, a contrastive learning method is used for training to obtain domain vectors representing the semantic features of different domains.

[0009] S3. Introduce the domain vector into a preset hybrid expert model, fuse the domain vector with the output of the activated expert network in the hybrid expert model, and train the hybrid expert model based on the result of the information fusion.

[0010] S4. The fused features obtained after information fusion are processed through the feature mapping layer to convert the fused features into a representation that matches the input format of the pre-trained language model main network, and then input into the computation layer of the main network for computation.

[0011] S5. Use domain-labeled data to train the hybrid expert model built in steps S1 to S4 to obtain the final domain task representation-enhanced MoE expert model, and use the model to infer the data to be predicted.

[0012] This invention significantly improves the adaptability and representation accuracy of large-scale models in specific domains by constructing a technical solution that deeply integrates task representation with the MoE architecture. Specifically, the method first establishes a broad knowledge base by collecting multi-domain data, and then uses contrastive learning to accurately extract domain vectors that characterize the unique semantic features of different domains, providing key guidance for the model to understand task differences. Subsequently, these domain vectors are introduced into a hybrid expert model and fused with the output of the activated expert network, achieving selective enhancement and dynamic reorganization of expert knowledge, enabling the model's output to be more focused on the core needs of the current domain. On this basis, a feature mapping layer transforms the fused features to match the input format of the main network, ensuring that domain knowledge can be seamlessly and smoothly injected into the subsequent computation of the pre-trained model, avoiding damage to the original model structure. Finally, the model obtained after the entire training process can achieve efficient expert routing based on domain prior knowledge, that is, automatically and accurately activate and combine the most relevant expert sub-networks for tasks in different domains during inference, thereby effectively alleviating the problem of insufficient representation of general models on specific tasks while maintaining the strong generalization ability of large-scale models, and greatly enhancing the model's representation ability and processing efficiency for domain tasks.

[0013] Preferably, in step S2, obtaining domain vectors capable of representing semantic features of different domains includes:

[0014] S21. Suppose there exists a domain set. For each field Initialize a corresponding neighborhood vector All domain vectors constitute a domain vector matrix. Where k represents the dimension of the neighborhood vector, and N represents the number of neighborhoods; the neighborhood vector The initialization uses random weights or pre-trained weights based on existing domain semantic embeddings, and updates them as learnable parameters during training.

[0015] S22, For each domain Input its corresponding training data The original features of the j-th sample are represented as follows: d represents the feature dimension of the original input sample, and a pre-trained language model is used as the encoder. The original input is mapped to a shared semantic space to generate corresponding semantic vector representations. , where θ is the encoder parameter;

[0016] S23. Construct positive and negative sample pairs for contrastive learning: for any training samples Its field is The semantic representation is To classify them as belonging to the same field The set of semantic representations corresponding to other samples that do not contain themselves is defined as the positive sample set. ; to bring together all those from different fields The semantic representation of the sample is defined as the negative sample set. .

[0017] By initializing a learnable domain vector for each domain and refining these vectors using contrastive learning, a set of domain vector representations capable of accurately characterizing the unique semantic features of different domains was successfully constructed. Specifically, this method first assigns an independent, trainable vector to each domain, enabling it to flexibly adapt to the semantic space of a specific domain. Then, a pre-trained language model maps the original input to a shared semantic space, ensuring that data from different domains can be compared based on unified features. Most importantly, by carefully constructing positive and negative sample pairs for contrastive learning—that is, bringing different samples within the same domain closer together in the semantic space to enhance intra-class compactness, while simultaneously widening the distance between samples from different domains to highlight inter-domain differences—the trained domain vectors are highly condensed and accurately reflect the core semantic features of their respective domains. These domain vectors, optimized based on contrastive learning, provide high-quality, highly discriminative domain prior knowledge for subsequent integration with the MoE architecture, serving as a key foundation for achieving efficient task-adaptive expert routing.

[0018] Preferably, in step S21, for each field The neighborhood vector A hierarchical domain representation structure is used for modeling, specifically representing a combination of global domain semantics and sample conditional local semantics:

[0019]

[0020] in, This is a global domain vector used to characterize the long-term stable and general knowledge features of the domain; This is a local domain vector used to represent the differences in different subtasks or semantic patterns within a domain; For input samples The semantic representation is adaptively generated adjustment coefficients, which are used to dynamically adjust the contribution of local domain semantics according to specific samples.

[0021] By decoupling domain vectors into a hierarchical combination of globally stable features and sample-adaptive local features, the granularity and flexibility of domain representation are significantly improved. Specifically, the global vector anchors general knowledge within the domain, while the local vector captures semantic differences within the domain, and the fusion weights of the two are dynamically adjusted by the samples. This design allows domain vectors to more accurately adapt to the semantic needs of specific samples, providing more expressive and adaptive prior information for subsequent expert routing and feature fusion.

[0022] Preferably, based on the positive and negative sample pairs constructed in step S23, a contrastive loss function is used to jointly optimize the encoder and the domain vector. The contrastive loss function is the enhanced domain-aware InfoNCE loss function. Its comparative loss Defined as:

[0023]

[0024] in, is the weighting coefficient, used to balance sample-level semantic consistency with domain-level semantic constraints; sim(⋅) is the similarity calculation function; To the positive sample set The positive samples selected from the data; For the field The corresponding domain vector; For temperature parameters; M represents the domain. The number of samples; N represents the number of domains; The first negative sample in the set l The first field n The semantic representation of each sample; The negative sample weights are used to characterize the difficulty of distinguishing the current sample from different negative samples, and are defined as follows:

[0025]

[0026] The overall training objective L is a weighted sum of the contrastive losses for all domains and all samples:

[0027] .

[0028] By designing an enhanced domain-aware InfoNCE loss function, joint optimization of the encoder and domain vectors was achieved, significantly improving the representation quality and training efficiency of contrastive learning. This loss function cleverly balances sample-level semantic consistency with domain-level semantic constraints through weight coefficients, enabling the model to both narrow the semantic distance between samples within the same domain and strengthen the alignment between samples and their respective domain vectors. Simultaneously, a similarity-weighted negative sample weighting mechanism is introduced to dynamically adjust the contribution of difficult and easy negative samples, allowing the model to focus more on distinguishing semantically confusing cross-domain samples. This dual optimization mechanism forces the encoder to extract more discriminative features and drives the domain vectors to more accurately condense the core semantics of the domain, ultimately obtaining high-quality domain representations with clear semantic boundaries, compact intra-class representations, and separation between classes. This lays a solid foundation for downstream expert routing and feature fusion tasks.

[0029] Preferably, a cross-domain soft boundary modeling mechanism is further introduced during the comparative learning process:

[0030] Define the domain similarity matrix Used to depict different fields and The semantic similarity between them; in the weighting of negative samples In the calculation process, the domain similarity matrix is... Introducing a weighted term, where, and It is positively correlated with the current field Domains with high semantic similarity Corresponding negative sample weights This relatively increases, thereby reducing the rejection strength of negative samples in the contrastive loss function.

[0031] By introducing a cross-domain soft-boundary modeling mechanism, the constraint strategy of contrastive learning is further optimized, making it more consistent with the continuity and hierarchy of the real semantic space. Specifically, this method quantifies the semantic proximity between different domains by constructing a domain similarity matrix and incorporates this similarity into the calculation of negative sample weights, achieving differentiated processing of negative samples from different sources: for semantically similar domains, samples from these domains generate a weaker repulsive force when used as negative samples, avoiding excessive separation between semantically adjacent domains; while for domains with significant semantic differences, stronger discriminative constraints are applied to ensure clear category boundaries. This soft-boundary design breaks the traditional hard-boundary assumption of "either positive or negative" in contrastive learning, enabling the model to more finely shape the topological relationships of the feature space while preserving the semantic association structure between domains. Ultimately, it obtains domain representations with natural semantic transitions and distinct hierarchies, providing more semantically insightful prior guidance for subsequent expert routing.

[0032] Preferably, step S3, which involves introducing the domain vector into the hybrid expert model, includes:

[0033] For input samples Encode it to obtain its semantic representation. Semantic representation of input samples through domain-aware gating functions. The activation scores of each expert are jointly modeled using their corresponding domain vector v(x).

[0034]

[0035] in, This indicates a feature concatenation operation. For the first The gating network function corresponding to each expert, For the first The activation scores of individual experts; by explicitly introducing domain semantics into the gating computation process, the expert selection process is endowed with domain awareness.

[0036] Based on the activation scores, the top-K experts with the highest scores are selected for activation, and their output representation matrices are obtained. , where K is the number of activated experts.

[0037] By concatenating the semantic representation of a sample with its corresponding hierarchical domain vector, and designing a domain-aware gating function based on this joint representation, a leapfrog upgrade of the expert selection mechanism from "general semantic-driven" to "domain semantic-driven" is achieved. This method enables the gating network to not only perceive the individual semantic features of the current sample when calculating the activation scores of each expert, but also to explicitly incorporate the global and local prior knowledge of its respective domain, thereby making more domain-specific routing decisions. The Top-K experts selected based on this domain-aware score selection have output representation matrices that more accurately focus on the professional knowledge subspace required for the current domain task, effectively avoiding the expert resource mismatch or knowledge interference problems that may occur with general routing mechanisms, and providing a purer and more suitable source of expert knowledge for subsequent domain-specific feature fusion.

[0038] Preferably, step S3, which involves introducing the domain vector into the hybrid expert model, further includes processing the output representation matrix. Perform layered fusion to obtain the final expert fusion output:

[0039] Each activated output is represented in the matrix. Its corresponding gating activation score Joint modeling is performed, and expert-level weight coefficients are calculated using a normalization function:

[0040]

[0041] in, This is the expert temperature parameter, used to adjust the smoothness of the expert weight distribution;

[0042] Based on the expert-level weighting coefficients The first-level weighted aggregation of the expert outputs is performed to obtain the initial expert fusion representation:

[0043]

[0044] Introducing a feature-level fusion mechanism, for each output representation matrix Dimension alignment and nonlinear mapping are performed to obtain the transformed representation. ,in For the first The learnable mapping matrix corresponding to each expert;

[0045] Semantic representation based on input samples After being changed by various experts The similarity between them is used to calculate feature-level attention weights through the softmax function:

[0046]

[0047] Where sim(⋅) is the similarity calculation function;

[0048] And based on the feature-level weights, perform two-layer fusion of the expert representations:

[0049]

[0050] Will and Joint integration is performed, and the final expert fusion output is obtained through residual connections and nonlinear transformations:

[0051]

[0052] in, This represents a vector concatenation operation. and For learnable linear transformation parameters, It is a non-linear activation function.

[0053] By constructing a hierarchical fusion mechanism to finely integrate activated expert outputs, the fusion quality and representation capability of multi-expert information are significantly improved. This method first performs a first-layer weighted aggregation of expert outputs based on gated activation scores, using expert-level weight coefficients to ensure that the knowledge of high-confidence experts dominates the fusion process. Subsequently, a feature-level fusion mechanism is introduced, dynamically capturing the most relevant feature dimensions of each expert output to the current task by calculating the similarity between sample semantics and the transformed representations of each expert, thus deepening the process from "expert selection" to "feature selection." Finally, residual connections and nonlinear transformations organically integrate the two-layer fusion results, enabling the model to both preserve the global contribution structure of experts and finely extract key local features within experts, providing high-quality domain-enhanced features for subsequent input to the main network.

[0054] Preferably, in obtaining the final expert fusion output Subsequently, the expert fusion representation is further adjusted at the feature level using domain vectors, including:

[0055] In obtaining the final expert fusion output Then, the mapping function is used to obtain the neighborhood vector. Generate inter-domain adjustment parameters and The following inter-domain adjustment operations are performed on the expert fusion representation:

[0056]

[0057] in, For the domain of the input sample The corresponding domain vector, This represents element-wise multiplication. For scaling parameters, This is the offset parameter.

[0058] By introducing a feature-level adjustment mechanism based on domain vectors after the expert fusion output, refined domain adaptation of the fused representation is achieved. Specifically, this method dynamically generates scaling and offset parameters using vectors from the domain to which the input samples belong, and performs an element-wise affine transformation on the expert fusion representation. This allows the model to adaptively strengthen or suppress different dimensions of the fused features according to domain semantic features and compensate for domain-specific distribution offsets. This post-processing adjustment mechanism is equivalent to applying a domain-specific "feature recalibration" operation to the general expert fusion result, ensuring that the feature representation of the final input main network not only structurally integrates multi-expert knowledge but also highly aligns with the specific domain requirements in terms of numerical distribution and semantic centroid, thereby further improving the model's representation accuracy and adaptability for domain tasks.

[0059] Preferably, the output represents the matrix. Defined as:

[0060]

[0061] in, For the first The forward propagation function of an expert network, To control the hyperparameters of the domain bias strength, For the introduction of the first Each expert has a learnable domain bias vector, which is used to characterize the domain preferences that the expert develops during long-term training.

[0062] By introducing learnable domain bias vectors into each expert network, this method endows experts with innate preferences and specializations in specific domains. During expert forward propagation, the semantic representation of the input sample is fused with the expert's own domain bias vector, enabling experts to actively activate their preferred domain knowledge patterns when processing samples, rather than passively relying on the gating network's selection. This design allows different experts to naturally differentiate their respective domain expertise during long-term training, forming a complementary expert cluster. Simultaneously, hyperparameters control the intensity of domain bias intervention, allowing the model to flexibly balance generality and specialization. Ultimately, this inherent domain awareness of experts and the externally gated domain routing mechanism work together to construct a complete domain-adaptive chain from sample to expert to fusion, significantly improving the model's specialization level and knowledge organization efficiency when handling multi-domain tasks.

[0063] The MoE expert model construction system based on domain task representation enhancement described in this invention includes:

[0064] The data collection module is used to collect domain-labeled data from several preset domains for training pre-trained language models.

[0065] The domain vector training module is used to train domain vectors based on collected domain-labeled data using a contrastive learning method to obtain domain vectors that represent the semantic features of different domains.

[0066] The information fusion and training module is used to introduce the domain vector into a preset hybrid expert model, fuse the domain vector with the output of the activated expert network in the hybrid expert model, and train the hybrid expert model based on the result of information fusion.

[0067] The feature mapping module is used to process the fused features obtained after information fusion through the feature mapping layer, convert the fused features into a representation that matches the input format of the pre-trained language model main network, and input it into the computation layer of the main network for calculation.

[0068] The model training and prediction module is used to train the constructed hybrid expert model using domain-labeled data to obtain the final domain task representation-enhanced MoE expert model, and to use the model to perform inference on the data to be predicted.

[0069] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. By introducing domain vectors into the gating and fusion mechanism of MoE, this invention enables the model to dynamically select and activate relevant experts based on the input semantics, achieving precise control of the computation path by domain knowledge, and significantly improving the adaptive ability of large-scale pre-trained models in multi-domain scenarios; 2. By using highly discriminative domain vectors to guide experts to focus on their areas of expertise for feature extraction, cross-domain knowledge interference is effectively avoided, and the generalization ability and inference reliability of the model on complex heterogeneous data are greatly improved; 3. Through hierarchical domain representation and expert bias design, static domain knowledge and dynamic data learning are organically combined, enabling the model to utilize structured priors and continuously optimize through data during training, ultimately constructing a unified model with excellent semantic understanding and task adaptation. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0071] Figure 2 This is a schematic diagram of the model structure of the present invention. Detailed Implementation

[0072] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0073] This invention provides a method for constructing a MoE expert model based on domain task representation enhancement. By comparing and learning between domain vectors and domain data, better domain vector content is obtained for initializing domain vectors. In the MoE structure, domain vector concatenation is added, and the network structure is modified. During model training and inference phases, the model gains a better understanding of domain knowledge, thereby further improving the model's inference performance. Figure 1 As shown, the method includes the following steps:

[0074] S1: Based on multiple pre-defined domains, systematically collect domain-related data to meet the training needs of large-scale models. Taking Nanjing government affairs as an example, for the 18 specific domains identified, obtain corresponding high-quality data samples, covering areas such as housing provident fund, real estate, culture and tourism, health care, social security, talent services, education, other government affairs, finance, law, tools, life, marriage, public welfare, public security, enterprises, transportation, and wallets, to ensure that the data fully represents the characteristics of each domain in terms of quantity and diversity, providing a solid foundation for subsequent model training.

[0075] S2: Data from different domains yields different results for large models. Domain vectors are added to the large model structure to assist in generating results. The specific process includes:

[0076] S21: Assume there exists a domain For each domain, initialize a corresponding domain vector. as follows:

[0077]

[0078] in, Represents the vector dimension. Indicates the number of domains.

[0079] The domain vector can be initialized using random weights or pre-trained weights based on existing domain semantic embeddings. This vector is continuously updated as a parameter during training, aiming to capture the semantic features of each domain, thereby assisting the model in effectively identifying and distinguishing information from different domains.

[0080] In a further implementation, for each domain The domain vector is no longer represented by a single static vector, but is modeled as a hierarchical domain representation structure that includes global domain semantics and sample conditional local semantics. Specifically, the domain vector... Defined as:

[0081]

[0082] in, Used to characterize long-term stable and general knowledge features in a field. Used to represent the differential features of different subtasks or semantic patterns within a domain. These are adjustment coefficients adaptively generated from the semantic representation of the input samples, used to dynamically adjust the contribution of local domain semantics based on specific samples. This hierarchical modeling approach enables domain representations to maintain domain consistency while possessing the ability to respond to changes in sample semantics.

[0083] S22: For each domain Input a set of training data The original features of the j-th sample are represented as follows: d represents the feature dimension of the original input sample, and a pre-trained BERT model is used as the encoder. The original input is mapped to a shared semantic space to generate a corresponding semantic vector representation. This shared semantic space ensures that the semantic representations of samples from different domains are comparable within the same vector space, facilitating subsequent comparative learning and domain differentiation. At the same time, the encoder parameters will be continuously optimized during training to improve the overall representation ability and generalization effect.

[0084] S23: For any training sample (Its field is) The semantic representation is ), will belong to the same field Furthermore, the set of semantic representations corresponding to other samples that do not contain themselves is defined as the positive sample set, i.e. These samples share the same domain semantic features; all samples from different domains The semantic representation corresponding to the sample is defined as the negative sample set, i.e. These samples represent semantic differences between domains. Positive and negative sample pairs are constructed based on these two types of samples to guide the model in contrastive learning to distinguish semantic representations between the same domain and different domains.

[0085] S24: In the contrastive learning phase, an enhanced domain-aware InfoNCE loss function is defined to jointly constrain the relationship between the sample semantic representation and the domain semantic representation. Specifically, let the... The domain vectors corresponding to each domain are: ,field The Middle The semantic vectors obtained by the encoder from each training sample are represented as follows: Define sample The contrast loss is:

[0086]

[0087] in, α is a weighting coefficient used to balance sample-level semantic consistency with domain-level semantic constraints. When set to 0, the model relies only on the semantic similarity between samples; when set to 1, the model relies only on the similarity between the sample and the domain vector. Experiments have shown that in 18 domains within the Nanjing government affairs scenario, when α is between 0.4 and 0.6, the model achieves the best balance between domain classification accuracy and downstream task performance. In this embodiment, α is preferably set to 0.5. The negative sample weights are used to characterize the difficulty of distinguishing the current sample from different negative samples, and are defined as follows:

[0088]

[0089] in τ is a temperature parameter used to adjust the logarithmic scale of the similarity distribution, affecting the model's ability to distinguish negative samples. A smaller τ value (approaching 0) results in a stronger repulsion towards negative samples semantically closest to the current sample, thus enhancing the discrimination of samples with blurred boundaries. A larger τ value (approaching 1) results in a smoother distribution of negative sample weights, with the model focusing on global discrimination of all negative samples. In this embodiment, considering the semantic overlap between some sub-domains of the government affairs field (such as "housing provident fund" and "real estate"), the range of τ is set to (0, 0.5], with a preferred value of 0.1, to strengthen boundary discrimination. Finally, the overall training objective is defined as the weighted sum of the contrastive losses for all domains and all samples:

[0090]

[0091] To avoid comparison conflicts between semantically similar domains, a cross-domain soft boundary modeling mechanism is further introduced during the comparative learning process. Specifically, a domain similarity matrix is ​​defined. This matrix is ​​used to characterize the semantic similarity between different domains. A value closer to 1 indicates greater semantic similarity between the two domains, while a value closer to 0 indicates greater difference. In this embodiment, the matrix can be initialized by calculating the cosine similarity of the center points of samples from each domain in the BERT encoding space. For example, the similarity between "social security" and "talent services" is calculated to be 0.75, while the similarity between "social security" and "transportation" is 0.15. During the calculation of negative sample weights, domain similarity is introduced into the weighting term. This results in a weaker repulsion constraint from negative samples from semantically similar domains and a stronger discriminative constraint from samples from semantically different domains. This effectively alleviates the semantic boundary ambiguity problem in multi-domain scenarios and improves the stability and discriminative power of domain vector learning.

[0092] S3: The domain vectors obtained in step S2 are fused with the output of the expert network activated by MoE to enhance the model's ability to express domain features. The model structure is as follows: Figure 2As shown.

[0093] The specific integration method is as follows:

[0094] S31: For the input sample Encode it to obtain its semantic representation. By jointly modeling the semantic representation h of the input sample and its corresponding domain vector v(x) using a domain-aware gating function, the activation scores of each expert are calculated:

[0095]

[0096] in, This represents the feature concatenation operation. By explicitly introducing domain semantics into the gating computation process, the expert selection process gains domain awareness, thereby achieving more stable expert routing that conforms to domain characteristics.

[0097] Then, the top-K experts are selected to obtain their output representation matrices. The most relevant expert network is selected and activated based on its weight.

[0098] S32: In a further implementation, to enhance the ability of different experts to model semantic patterns in a specific domain, a domain bias vector is introduced into each expert network. This is used to characterize the expert's domain preferences developed during long-term training. The output represents the matrix... Defined as:

[0099]

[0100] in, To control the hyperparameter of domain bias strength, when λ=0, the expert network degenerates into a general expert, not reflecting domain preference; the larger λ is, the stronger the domain specificity of the expert output. To avoid overfitting to a specific domain and thus reducing generalization ability, this embodiment restricts λ to the range [0, 1], preferably 0.2, so that the expert can moderately enhance the response to specific domain patterns while maintaining general ability. This design enables different experts to form a stable domain function division while sharing the overall structure, improving expert utilization and routing consistency. Subsequently, hierarchical fusion is performed on the output results of the Top-K experts activated in step S31. Specifically, the following process is included: First, the output representation of each activated expert network is... Its corresponding gate score Joint modeling is performed, and expert-level weight coefficients are calculated using a normalization function:

[0101]

[0102] in, This is the expert temperature parameter, used to adjust the smoothness of the expert weight distribution. When the weights approach 0, the weight distribution tends to a one-hot form, activating only the expert with the highest score (strong sparsity of routing); when As the number of experts increases, the weight distribution tends to become more uniform, and more experts are smoothly activated (weak sparsity in routing). In the Top-2 expert selection strategy of this embodiment, in order to retain a certain degree of expert diversity while ensuring the dominance of the main experts, the weight distribution is adjusted accordingly. The preferred value is 1.0. Subsequently, based on the expert-level weights, a first-level weighted aggregation is performed on the outputs of each expert to obtain the initial expert fusion representation:

[0103]

[0104] S33: Building upon this, to further characterize the differences among experts at the feature dimension level, a feature-level fusion mechanism is introduced to output for each expert. Dimension alignment and nonlinear mapping are performed to obtain the transformed representation. Subsequently, semantic representation based on the input samples... Calculate feature-level attention weights based on the similarity between the outputs of each expert:

[0105]

[0106] And a second layer of fusion is performed on the expert representation based on the aforementioned feature-level weights:

[0107]

[0108] Finally, and Joint integration is performed, and the final expert fusion output is obtained through residual connections and nonlinear transformations:

[0109]

[0110] in, This represents a vector concatenation operation. It is a non-linear activation function.

[0111] After obtaining the final expert fusion representation *e*, the expert representation is further adjusted at the feature level using the domain vector. Specifically, adjustment parameters are generated from the domain vector through a mapping function. and The following adjustments were made to the expert fusion approach:

[0112]

[0113] in, This represents element-wise multiplication. Through this adjustment mechanism, domain knowledge is applied parametrically to the expert fusion representation, thereby enhancing the model's ability to express domain features without significantly increasing computational overhead.

[0114] Through the above-mentioned hierarchical fusion process, it is possible not only to distinguish the overall contribution of different experts to the input sample at the expert level, but also to characterize the complementary relationship between expert outputs at the feature dimension level, thereby providing a more sufficient and stable expert representation basis for the subsequent dynamic fusion of domain vectors.

[0115] S4: The fused features output by the MoE module are converted into a representation that matches the input format of the large model through the feature mapping layer, and then passed as input to the subsequent computation layers of the large model to ensure the integrity and availability of information during cross-module transfer.

[0116] S5: Train the model from S4 using domain data to obtain a new model, then use the new data to predict results, ultimately achieving better performance.

[0117] S6: To verify the effectiveness of this invention in real-world multi-domain scenarios, experiments were conducted using data from the "Housing Provident Fund" and "Social Security" domains within the Nanjing government service scenario. First, the model trained in steps S1–S5 was used to perform inference tests in the aforementioned two domains. Experimental metrics included routing accuracy, inference speed, and the effectiveness of dynamic weight adjustment for domain vectors.

[0118] Routing accuracy: Under the Top-K expert selection and multi-stage domain awareness fusion mechanism, the correct expert routing rate of the "housing provident fund" sample reached 95.3%, and that of the "social security" sample reached 94.7%, which is about 4-6 percentage points higher than the traditional MoE + fixed weighted fusion method.

[0119] Inference speed: Under the expert activation strategy of maintaining Top-K=4, the average inference time per sample is 23ms / sample, which has almost no additional computational overhead compared with the traditional splicing and fusion method, ensuring practical usability.

[0120] Dynamic adjustment of domain weights: Within the same domain, the contribution rate of the domain vectors of different input samples is adjusted by... Automatic adjustment ensures that the domain vector weights of highly semantically relevant samples are significantly higher than those of low-relevance samples under given sample conditions, achieving sample-level adaptive feature enhancement and improving the model's ability to distinguish fine-grained tasks.

[0121] To further verify the causal role of domain vectors in expert routing and inference processes, this invention designed a counterfactual control experiment. While keeping the input samples unchanged, the domain vectors were subjected to fixed, perturbed, and randomly replaced processing, respectively. Experimental results show that when the domain vectors are perturbed or randomly replaced, both expert routing consistency and inference accuracy significantly decrease, verifying that domain vectors do not merely exist as auxiliary features in the model inference process, but rather have a substantial impact on expert selection and feature enhancement.

[0122] Experimental results show that the multi-stage domain-aware MoE fusion mechanism proposed in this invention not only improves the accuracy of expert routing and tasks, but also achieves efficient reasoning and domain semantic adaptation capabilities in actual government scenarios, providing a feasible technical solution for the application of large models in multi-domain government affairs.

[0123] Based on a similar inventive concept, embodiments of the present invention also provide a MoE expert model construction system based on domain task representation enhancement, corresponding to the method for constructing MoE expert models based on domain task representation enhancement, comprising:

[0124] The data collection module is used to collect domain-labeled data from several preset domains for training pre-trained language models.

[0125] The domain vector training module is used to train domain vectors based on collected domain-labeled data using a contrastive learning method to obtain domain vectors that represent the semantic features of different domains.

[0126] The information fusion and training module is used to introduce the domain vector into a preset hybrid expert model, fuse the domain vector with the output of the activated expert network in the hybrid expert model, and train the hybrid expert model based on the result of information fusion.

[0127] The feature mapping module is used to process the fused features obtained after information fusion through the feature mapping layer, convert the fused features into a representation that matches the input format of the pre-trained language model main network, and input it into the computation layer of the main network for calculation.

[0128] The model training and prediction module is used to train the constructed hybrid expert model using domain-labeled data to obtain the final domain task representation-enhanced MoE expert model, and to use the model to perform inference on the data to be predicted.

Claims

1. A method for constructing a MoE expert model based on domain task representation enhancement, characterized in that, Includes the following steps: S1. Collect several pre-defined data with domain labels from different domains for training a pre-trained language model; S2. Based on the domain-labeled data collected in step S1, a contrastive learning method is used for training to obtain domain vectors representing the semantic features of different domains. S3. Introduce the domain vector into a preset hybrid expert model, fuse the domain vector with the output of the activated expert network in the hybrid expert model, and train the hybrid expert model based on the result of the information fusion. S4. The fused features obtained after information fusion are processed through the feature mapping layer to convert the fused features into a representation that matches the input format of the pre-trained language model main network, and then input into the computation layer of the main network for computation. S5. Use domain-labeled data to train the hybrid expert model built in steps S1 to S4 to obtain the final domain task representation-enhanced MoE expert model, and use the model to infer the data to be predicted.

2. The method according to claim 1, characterized in that, In step S2, obtaining domain vectors representing semantic features of different domains includes: S21. Suppose there exists a domain set. For each field Initialize a corresponding neighborhood vector All domain vectors constitute a domain vector matrix. Where k represents the dimension of the neighborhood vector, and N represents the number of neighborhoods; the neighborhood vector The initialization uses random weights or pre-trained weights based on existing domain semantic embeddings, and updates them as learnable parameters during training. S22, For each domain Input its corresponding training data The original features of the j-th sample are represented as follows: d represents the feature dimension of the original input sample, and a pre-trained language model is used as the encoder. The original input is mapped to a shared semantic space to generate corresponding semantic vector representations. , where θ is the encoder parameter; S23. Construct positive and negative sample pairs for contrastive learning: for any training samples Its field is The semantic representation is To classify them as belonging to the same field The set of semantic representations corresponding to other samples that do not contain themselves is defined as the positive sample set. ; to bring together all those from different fields The semantic representation of the sample is defined as the negative sample set. .

3. The method according to claim 2, characterized in that, In step S21, for each field The neighborhood vector A hierarchical domain representation structure is used for modeling, specifically representing a combination of global domain semantics and sample conditional local semantics: ;in, This is a global domain vector used to characterize the long-term stable and general knowledge features of the domain; This is a local domain vector used to represent the differences in different subtasks or semantic patterns within a domain; For input samples The semantic representation is adaptively generated adjustment coefficients, which are used to dynamically adjust the contribution of local domain semantics according to specific samples.

4. The method according to claim 2, characterized in that, Based on the positive and negative sample pairs constructed in step S23, a contrastive loss function is used to jointly optimize the encoder and the domain vector. This contrastive loss function is the enhanced domain-aware InfoNCE loss function. For each sample... Its comparative loss Defined as: ;in, is the weighting coefficient, used to balance sample-level semantic consistency with domain-level semantic constraints; sim(⋅) is the similarity calculation function; To the positive sample set The positive samples selected from the data; For the field The corresponding domain vector; For temperature parameters; M represents the domain. The number of samples; N represents the number of domains; The first negative sample in the set l The first field n The semantic representation of each sample; The negative sample weights are used to characterize the difficulty of distinguishing the current sample from different negative samples, and are defined as follows: The overall training objective L is a weighted sum of the contrastive losses for all domains and all samples: 。 5. The method according to claim 4, characterized in that, A cross-domain soft boundary modeling mechanism is further introduced during the comparative learning process: Define the domain similarity matrix Used to depict different fields and The semantic similarity between them; in the weighting of negative samples In the calculation process, the domain similarity matrix is... Introducing a weighted term, where, and It is positively correlated with the current field Domains with high semantic similarity Corresponding negative sample weights This relatively increases, thereby reducing the rejection strength of negative samples in the contrastive loss function.

6. The method according to claim 1, characterized in that, Step S3, which involves introducing the domain vector into the hybrid expert model, includes: For input samples Encode it to obtain its semantic representation. Semantic representation of input samples through domain-aware gating functions. The activation scores of each expert are jointly modeled using their corresponding domain vector v(x). ;in, This indicates a feature concatenation operation. For the first The gating network function corresponding to each expert, For the first The activation scores of individual experts; by explicitly introducing domain semantics into the gating computation process, the expert selection process is endowed with domain awareness. Based on the activation scores, the top-K experts with the highest scores are selected for activation, and their output representation matrices are obtained. , where K is the number of activated experts.

7. The method according to claim 6, characterized in that, Step S3, which involves introducing the domain vector into the hybrid expert model, also includes processing the output representation matrix. Perform layered fusion to obtain the final expert fusion output: Each activated output is represented in the matrix. Its corresponding gating activation score Joint modeling is performed, and expert-level weight coefficients are calculated using a normalization function: ;in, This is the expert temperature parameter, used to adjust the smoothness of the expert weight distribution; Based on the expert-level weighting coefficients The first-level weighted aggregation of the expert outputs is performed to obtain the initial expert fusion representation: A feature-level fusion mechanism is introduced to fuse each output representation matrix. Dimension alignment and nonlinear mapping are performed to obtain the transformed representation. ,in For the first The learnable mapping matrix corresponding to each expert; Semantic representation based on input samples After being changed by various experts The similarity between them is used to calculate feature-level attention weights through the softmax function: ; where sim(⋅) is the similarity calculation function; And based on the aforementioned feature-level weights, perform two-layer fusion of the expert representations: ;Will and Joint integration is performed, and the final expert fusion output is obtained through residual connections and nonlinear transformations: ;in, This represents a vector concatenation operation. and For learnable linear transformation parameters, It is a non-linear activation function.

8. The method according to claim 7, characterized in that, In obtaining the final expert fusion output Subsequently, the expert fusion representation is further adjusted at the feature level using domain vectors, including: In obtaining the final expert fusion output Then, the mapping function is used to obtain the neighborhood vector. Generate inter-domain adjustment parameters and The following inter-domain adjustment operations are performed on the expert fusion representation: ;in, For the domain of the input sample The corresponding domain vector, This represents element-wise multiplication. For scaling parameters, This is the offset parameter.

9. The method according to claim 6, characterized in that, The output representation matrix The expression is: ;in, For the first The forward propagation function of an expert network, To control the hyperparameters of the domain bias strength, For the introduction of the first Each expert has a learnable domain bias vector, which is used to characterize the domain preferences that the expert develops during long-term training.

10. A MoE expert model construction system based on domain task representation enhancement, characterized in that, include: The data collection module is used to collect domain-labeled data from several preset domains for training pre-trained language models. The domain vector training module is used to train domain vectors based on collected domain-labeled data using a contrastive learning method to obtain domain vectors that represent the semantic features of different domains. The information fusion and training module is used to introduce the domain vector into a preset hybrid expert model, fuse the domain vector with the output of the activated expert network in the hybrid expert model, and train the hybrid expert model based on the result of information fusion. The feature mapping module is used to process the fused features obtained after information fusion through the feature mapping layer, convert the fused features into a representation that matches the input format of the pre-trained language model main network, and input it into the computation layer of the main network for calculation. The model training and prediction module is used to train the constructed hybrid expert model using domain-labeled data to obtain the final domain task representation-enhanced MoE expert model, and to use the model to perform inference on the data to be predicted.

Images