Token-aware expert hybrid model construction method and device based on pre-trained model, equipment and medium

Abstract

This application provides a method, apparatus, device, and medium for constructing a token-aware expert hybrid model based on a pre-trained model, relating to the field of artificial intelligence technology. The method includes: acquiring a calibration dataset; inputting the calibration dataset into a pre-trained dense language model, extracting intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset; determining the importance of each neuron in the feedforward network layer based on the intermediate activation features; dividing the neurons in the feedforward network layer into multiple non-overlapping neuron sets according to the importance of the neurons, obtaining a sparse hybrid expert model; for the input token to be processed, selecting at least one target expert from multiple experts based on the correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model; and processing the input token using at least one target expert to obtain the output result corresponding to the input token.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for constructing a token-aware expert hybrid model based on a pre-trained model. Background Technology

[0002] With the development of the Transformer architecture (a deep neural network model based on a self-attention mechanism), large-scale pre-trained language models (LLMs) have demonstrated outstanding performance on numerous natural language processing tasks. However, the dramatic increase in the number of model parameters has led to high computational costs and storage overhead, severely restricting their efficient inference and deployment in resource-constrained environments.

[0003] To alleviate this problem, the Mixture-of-Experts (MoE) architecture has been proposed. This architecture replaces some dense layers with multiple small expert networks and dynamically activates a few of these experts, achieving computational sparsity while maintaining model capacity. Related technical solutions typically involve two steps: first, the dense layers of the pre-trained model (such as feedforward network layers) are randomly or clustered into multiple experts; then, a learnable routing network is introduced, and continuous pre-training is performed using a large amount of data to learn the token-to-expert allocation strategy.

[0004] However, such methods have significant drawbacks: on the one hand, random or simple clustering splitting methods cannot ensure that experts possess differentiated knowledge structures, leading to expert homogenization and weakening the expert specialization advantage that MoE should have; on the other hand, introducing learnable routing networks brings additional training overhead and parameter burden, and is prone to overfitting or convergence instability problems in downstream task adaptation, limiting its feasibility and practicality in real-world applications. Therefore, how to efficiently construct high-performance sparse MoE models from existing dense models is an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and medium for constructing a token-aware expert hybrid model based on a pre-trained model. This solution can losslessly convert a pre-trained dense language model into a sparse hybrid expert model without introducing additional routing parameters, thus solving the problems of high training cost, low deployment efficiency, and difficulty in adapting to downstream tasks caused by the homogenization of expert knowledge and reliance on learnable routing networks in related methods.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a method for constructing a token-aware expert hybrid model based on a pre-trained model, the method comprising: Obtain a calibration dataset, wherein the sample data of the calibration dataset is selected from at least one of the following application scenarios: intelligent question answering, dialogue interaction, text summarization, and language translation text; The calibration dataset is input into a pre-trained dense language model, and intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset are extracted. Based on the intermediate activation features, the importance of each neuron in the feedforward network layer is determined; Based on the importance of neurons, the neurons in the feedforward network layer are divided into multiple non-overlapping sets of neurons, and each set of neurons constitutes an expert, resulting in a sparse hybrid expert model. For an input token to be processed, at least one target expert is selected from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model; wherein the expert features are characterized based on the response of the subset of neurons constituting the expert to the calibration dataset; The input token is processed using at least one target expert to obtain the output result corresponding to the input token.

[0007] Optionally, based on the intermediate activation features, the importance of each neuron in the feedforward network layer is determined, including: The intermediate activation features are clustered to obtain multiple feature clusters; wherein the number of feature clusters is set to a preset number of experts, and each feature cluster corresponds to one expert; For each feature cluster, the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster is analyzed to determine the importance of that neuron.

[0008] Optionally, for each feature cluster, the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster is analyzed, including: Calculate the norm of the weight vector corresponding to the neuron as the contribution; or, The contribution is calculated as the statistical measure of the activation value of the neuron when processing intermediate activation features belonging to the feature cluster.

[0009] Optionally, based on the importance of neurons, the neurons in the feedforward network layer are divided into multiple non-overlapping sets of neurons, including: For each expert to be constructed, based on the importance of the neurons, a predetermined number of neurons with the highest importance ranking are selected from the feedforward network layer and assigned to the neuron set corresponding to that expert. The sparse hybrid expert model includes a shared expert and multiple specialized experts. The shared expert processes all input tokens, while the specialized experts process a subset of the input tokens.

[0010] Optionally, for the input token to be processed, at least one target expert is selected from multiple experts based on the correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model, including: The input token is input to the shared expert, and the activation value of the input token in the shared expert is obtained as the token feature of the input token; For each specialized expert, obtain the activation values ​​of the neurons assigned to that specialized expert when processing the calibration dataset, and determine the expert features of that specialized expert based on the activation values; Calculate the degree of association between the token feature and each of the expert features; Based on the degree of correlation, at least one target expert is selected from the plurality of specialized experts.

[0011] Optionally, the degree of correlation can be characterized by Euclidean distance or cosine similarity.

[0012] Optionally, the input token is processed using the at least one target expert to obtain the output result corresponding to the input token, including: The input token is input to the shared expert to obtain the output vector of the shared expert, and the input token is input to the at least one target expert to obtain the output vector of each target expert; The aggregation weight corresponding to each target expert is determined based on the degree of correlation between the token features of the input token and the expert features of each target expert. Based on the aggregation weights, the output vectors of the at least one target expert and the output vectors of the shared expert are weighted and fused to obtain the output result corresponding to the input token.

[0013] A second aspect of this application discloses a token-aware expert hybrid model construction apparatus based on a pre-trained model, the apparatus comprising: The acquisition module is used to acquire a calibration dataset, wherein the sample data of the calibration dataset is selected from at least one of the following application scenarios: intelligent question answering, dialogue interaction, text summarization, and language translation text; An extraction module is used to input the calibration dataset into a pre-trained dense language model and extract intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset. A determination module is used to determine the importance of each neuron in the feedforward network layer based on the intermediate activation features; The partitioning module is used to divide the neurons in the feedforward network layer into multiple non-overlapping sets of neurons according to the importance of the neurons, and each set of neurons constitutes an expert, thus obtaining a sparse hybrid expert model. A selection module is used to select at least one target expert from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model, for the input token to be processed; wherein the expert features are characterized based on the response of the subset of neurons constituting the expert to the calibration dataset; The processing module is used to process the input token using the at least one target expert to obtain the output result corresponding to the input token.

[0014] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model described in the first aspect of this application.

[0015] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model described in the first aspect of this application.

[0016] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model as described in the first aspect of this application.

[0017] The embodiments of this application have the following advantages: In this embodiment, by utilizing calibration datasets from target application scenarios (such as intelligent question answering and text translation), intermediate activation features of pre-trained dense language models are extracted to determine the importance of neurons, and expert segmentation is performed based on the importance of neurons. Thus, without relying on large-scale continuous training, the original dense language model is automatically and losslessly converted into a sparse hybrid expert model, avoiding the training overhead and parameter burden brought about by introducing additional learnable routing networks, and reducing the computational and time costs of model compression and adaptation.

[0018] Furthermore, by classifying experts based on the importance of neurons, each expert is composed of a set of neurons with significant functional tendencies, thus achieving ordered differentiation of knowledge at the model structure level and constructing more discriminative expert features. In the inference stage, by matching input token features with expert features without parameters, it is possible to more accurately assign appropriate experts to tokens with different semantics, thereby improving the model's ability to process complex language patterns and the quality of its output.

[0019] Since expert features are pre-statically represented based on responses from a calibration dataset, and routing decisions depend on the correlation between input token features and expert features, no trainable parameters are required. This avoids overfitting or convergence instability issues that may arise from fine-tuning the routing network when deploying the model to different downstream application scenarios (such as intelligent question answering and text translation). A high-performance sparse model can be obtained with only one calibration and transformation based on the target application scenario, improving the practicality and cross-scenario transferability of the method. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of a token-aware expert hybrid model construction method based on a pre-trained model, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of an expert decomposition method for neuron importance provided in an embodiment of this application; Figure 3 This is a schematic diagram of a representation-based parameterless routing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of a token-aware expert hybrid model construction device based on a pre-trained model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a token-aware expert hybrid model construction method based on a pre-trained model, as provided in an embodiment of this application. Figure 1 As shown, the method for constructing a token-aware expert hybrid model based on a pre-trained model may include steps S110 to S160: Step S110: Obtain a calibration dataset, wherein the sample data of the calibration dataset is selected from at least one of the following application scenarios: intelligent question answering, dialogue interaction, text summarization, and language translation text.

[0024] The calibration dataset, which reflects the data distribution of the target application scenario, is smaller than the pre-training data. Its purpose is to provide representative data samples for feature analysis and structural partitioning in subsequent model conversion, without the need for large-scale training. For example, thousands to tens of thousands of application scenarios (question-answer pairs, translation sentence pairs) can be collected as calibration data.

[0025] Step S120: Input the calibration dataset into a pre-trained dense language model and extract the intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset.

[0026] The pre-trained dense language model can be a large language model based on the Transformer architecture, such as LLaMA (Large Language Model Meta AI) and GPT (Generative Pre-trained Transformer) series models. Intermediate activation features refer to the internal representation vector corresponding to each input token in the feedforward network layer after the calibration dataset is input into the pre-trained dense language model.

[0027] Specifically, the calibration dataset is input into a pre-trained dense language model. During the forward propagation process, the intermediate activation values ​​generated by at least one feed-forward network (FFN) layer within the dense language model for each token in the calibration data are captured. These activation values ​​constitute the intermediate activation features for each token at that layer. The dimension of these intermediate activation features is typically equal to the hidden layer dimension of the feed-forward network layer and reflects the internal state of the model at that feed-forward network layer when processing different semantic content.

[0028] Step S130: Based on the intermediate activation features, determine the importance of each neuron in the feedforward network layer.

[0029] The importance of neurons is used to quantify the differences in contributions of different neurons when processing the semantic patterns represented by the current calibration data. Specifically, based on the intermediate activation features of all tokens, statistical analysis or modeling methods can be used to evaluate the contribution of each neuron to the overall feature expression or to a specific subset of features, thereby assigning it an importance score.

[0030] Step S140: Based on the importance of neurons, the neurons in the feedforward network layer are divided into multiple non-overlapping sets of neurons, and each set of neurons constitutes an expert, thus obtaining a sparse hybrid expert model.

[0031] In this model, "experts" refer to subnetworks composed of specific combinations of neurons, separated from the feedforward network layer. Each expert is designed with different knowledge processing tendencies. In practice, for each expert to be constructed, the top K most important neurons from all neurons in the feedforward network layer are selected based on their importance ranking and assigned to the corresponding set. For example, to construct 8 experts, a total of 8 × K core neurons are selected, and each expert receives K of the most representative neurons. All experts together constitute a sparse hybrid expert model, replacing the original dense layer.

[0032] Step S150: For the input token to be processed, select at least one target expert from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model; wherein the expert features are characterized based on the response of the subset of neurons constituting the expert to the calibration dataset.

[0033] Specifically, during inference, the token features of the input token are first obtained (e.g., extracted through a base module or shared experts). Simultaneously, each expert has a pre-computed, fixed set of expert features. The correlation (e.g., similarity or distance) between the token features and each expert's features is calculated, and one or more experts with the highest correlation are dynamically selected as the target experts. This routing process requires no trainable parameters.

[0034] Step S160: Process the input token using the at least one target expert to obtain the output result corresponding to the input token.

[0035] Specifically, the input token is input to each target expert, and each target expert calculates the input token based on its own set of neurons. Finally, the processing results of these experts are integrated to generate the output result corresponding to the input token.

[0036] It is understood that the dense language model in this embodiment already possesses powerful general-purpose language understanding and generation capabilities, enabling it to support a wide range of natural language processing tasks. Using the method of this application, it can be efficiently reconstructed into a sparse hybrid expert model, thereby achieving significant performance and efficiency improvements in multiple application scenarios.

[0037] The pre-trained dense language model serves as the foundational model and can be directly or finely tuned for applications such as intelligent content generation, knowledge-based question answering and information retrieval, sentiment analysis and content security, educational assessment and language learning, and intelligent code assistance. The sparse hybrid expert model constructed using the method described in this application, while inheriting fundamental capabilities, exhibits significant advantages due to its unique sparse computation and dynamic expert routing mechanism in scenarios with higher requirements for real-time performance, resource efficiency, and task specialization: high-concurrency dialogue interaction, domain-adaptive machine translation, specialized text summarization and creation, accurate code generation and diagnosis, real-time recommendation and search enhancement, and edge computing and lightweight deployment.

[0038] As can be seen from the above applications, this application not only provides an efficient model conversion method, but also provides technical support for the implementation of large-scale pre-trained language models in diverse and efficient practical applications by introducing a sparse hybrid expert architecture.

[0039] The technical solution adopted in this application utilizes calibration datasets from target application scenarios (such as intelligent question answering and text translation) to extract intermediate activation features from a pre-trained dense language model, thereby determining the importance of neurons and performing expert segmentation based on neuron importance. This automatically and losslessly transforms the original dense language model into a sparse hybrid expert model without relying on large-scale retraining, avoiding the training overhead and parameter burden brought by introducing additional learnable routing networks, and reducing the computational and time costs of model compression and adaptation. Furthermore, segmenting experts based on neuron importance ensures that each expert consists of a set of neurons with significant functional biases, achieving ordered differentiation of knowledge at the model structure level and constructing more discriminative expert features. During the inference phase, by parameter-free matching of input token features and expert features, appropriate experts can be more accurately assigned to tokens with different semantics, thereby improving the model's ability to process complex language patterns and the quality of its output. Since expert features are pre-statically represented based on responses from a calibration dataset, and routing decisions depend on the correlation between input token features and expert features, no trainable parameters are required. This avoids overfitting or convergence instability issues that may arise from fine-tuning the routing network when deploying the model to different downstream application scenarios (such as intelligent question answering and text translation). A high-performance sparse model can be obtained with only one calibration and transformation based on the target application scenario, improving the practicality and cross-scenario transferability of the method.

[0040] In an optional embodiment, the importance of each neuron in the feedforward network layer is determined based on the intermediate activation features, including the following steps S130-1 to S130-2: Step S130-1: Cluster the intermediate activation features to obtain multiple feature clusters; wherein the number of feature clusters is set to a preset number of experts, and each feature cluster corresponds to one expert.

[0041] Clustering algorithms (such as K-means, spectral clustering, and Gaussian mixture models) are used to cluster the intermediate activation features extracted in step S120. The number of clusters is set to a preset number of experts. Clustering summarizes the implicit semantic patterns in the data in the feature space. After clustering, the intermediate activation features of all tokens are divided into multiple different feature clusters. Each feature cluster can be regarded as a natural aggregation of a semantic pattern or knowledge type in the model activation space. Each feature cluster will subsequently correspond to an expert to be constructed, thus initially establishing the correspondence between experts and semantic patterns in the structural design.

[0042] In some embodiments, one shared expert and n dedicated experts are preset, where the one shared expert corresponds to the intermediate activation feature of all tokens, and the n dedicated experts correspond to n different feature clusters. Assume... If the intermediate activation feature of the token is d, then the intermediate activation feature obtained by the shared expert is: N is the number of tokens in the calibration dataset, and the intermediate activation feature obtained by the i-th specialized expert is... The activation features belonging to the i-th feature cluster are used Let j represent (the value of j is 0 to 1). , (The number of activated features within a feature cluster).

[0043] Step S130-2: For each feature cluster, analyze the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster, as a measure of the importance of that neuron.

[0044] Here, contribution refers to a quantitative measure of a neuron's role in processing tokens belonging to a certain feature cluster (i.e., a certain semantic pattern). Specifically, the following analysis is performed for each feature cluster: First, all intermediate activation features corresponding to tokens belonging to that feature cluster are selected to form a feature subset of that cluster; then, for each neuron in the feedforward network layer, its contribution to processing that feature subset (such as weight norm or average activation intensity) is analyzed; finally, the contribution calculated above is used as the importance of that neuron relative to the current feature cluster (i.e., the future corresponding expert).

[0045] Optionally, step S130-2 above, "for each feature cluster, analyze the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster," may include steps S130-2-1 and S130-2-2, which are parallel optional solutions: Step S130-2-1: Calculate the norm of the weight vector corresponding to the neuron as the contribution.

[0046] In this context, the weight vector refers to the set of learnable parameters that each neuron (usually corresponding to a hidden unit) in the feedforward network layer has, used to receive input from the previous layer and calculate its weighted sum. The norm of the weight vector can be either the L1 norm (the sum of the absolute values ​​of the elements of the weight vector) or the L2 norm (the square root of the sum of the squares of the elements of the weight vector, i.e., the Euclidean length).

[0047] For each neuron in the feedforward network layer, its complete weight vector is obtained, and a specific norm of this weight vector is calculated. Since the norm of the weight vector in a neural network reflects the inherent strength of neuronal connections, neurons with larger norms generally have larger absolute parameter values ​​and are usually considered to play a more crucial role in feature transformation, thus contributing more to processing their associated feature clusters. Therefore, the norm of the weight vector corresponding to the neuron is used as the contribution.

[0048] Step S130-2-2: Calculate the statistics of the activation values ​​of the neuron when processing intermediate activation features belonging to the feature cluster as the contribution.

[0049] Specifically, for the currently analyzed feature cluster, firstly, all tokens belonging to that cluster are collected, and the activation values ​​of these tokens flowing through that specific neuron during forward propagation are obtained, thus forming a set of activation values. Then, a selected statistic is calculated for this set of activation values. For example: calculating the mean (reflecting the average response level of the neuron to the features of that cluster), calculating the maximum value (reflecting the maximum possible response strength of the neuron to the features of that cluster), and calculating the variance or standard deviation (which can reflect the specificity or consistency of the neuron's response). The calculated statistic reflects the actual response strength of the neuron to a specific data pattern and is used as the neuron's contribution to processing the current feature cluster.

[0050] Thus, computation can be performed using existing model parameters (weights) or intermediate results (activation values) obtained from a single forward propagation, eliminating the need for complex backpropagation or second-order optimization. This approach is computationally efficient and easy to implement. By selecting different contribution metrics, the expert construction process can be guided to focus on different characteristics (such as structural importance or data response characteristics). Consequently, the final set of neurons (experts) may possess different functional tendencies, further enriching the diversity of experts within the sparse hybrid expert model and improving the overall model's expressive power and routing accuracy.

[0051] The technical solution adopted in this embodiment directly correlates the implicit semantic patterns of the calibration data with the number of experts through clustering. This allows the construction of experts to be based on the inherent structure of the data distribution, laying a solid foundation for the subsequent formation of discriminative experts. For each expert, a clear and computable contribution evaluation method is provided for its candidate neurons, which provides an objective and operable standard for subsequently dividing the neuron set according to importance. Furthermore, each expert is associated with a feature cluster, and its neurons are selected based on their contribution to that cluster. The final constructed experts tend to be good at processing the input of their corresponding semantic patterns, making the internal working mechanism of the entire sparse hybrid expert model more interpretable.

[0052] In an optional embodiment, step S140 above, "dividing the neurons in the feedforward network layer into multiple non-overlapping sets of neurons according to their importance," may specifically include: For each expert to be constructed, based on the importance of the neurons, a predetermined number of neurons with high importance ranking are selected from the feedforward network layer and assigned to the neuron set corresponding to that expert. The sparse hybrid expert model includes one shared expert and multiple specialized experts. The shared expert processes all input tokens, and the specialized experts process a subset of the input tokens.

[0053] In this embodiment, all neurons in the feedforward network layer are ranked in descending order of their importance score relative to the expert currently being built. Then, starting from the top of the ranking list, a fixed number (e.g., K) of neurons are selected. This number K is preset and determines the size of each expert (i.e., the number of neurons it contains). These K neurons are then assigned to the expert currently being built (shared expert or dedicated expert), forming a neuron set specific to that expert. This process ensures that each neuron is assigned based on its maximum contribution to the target expert.

[0054] In a sparse hybrid expert model, a shared expert is a special expert that processes all input tokens. Its set of neurons is also selected from the feedforward network layers based on the importance of the neurons. Shared experts provide basic, general processing capabilities, ensuring that the sparse hybrid expert model has basic transformation functions for any input. Specialized experts, on the other hand, are experts in a sparse hybrid expert model that are activated only when the semantic features (token features) of the input token highly match their own. Each specialized expert consists of a set of functionally specialized neurons, adept at processing a specific type of semantic pattern.

[0055] It is understandable that for matrices containing increasing dimensions ( ) and dimensionality reduction matrix ( For a feedforward network layer with two dense matrices, each expert (shared expert, dedicated expert) obtained through the above method also has a different upscaling matrix. ) and dimensionality reduction matrix ( )matrix.

[0056] like Figure 2 As shown, Figure 2This is a schematic diagram of an expert decomposition method for neuron importance provided in an embodiment of this application. Through the decomposition method described in steps S110 to S140, a dense language model can be structurally transformed into a sparse hybrid expert model. Specifically, a calibration dataset is input into a pre-trained dense language model, and intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset are extracted. Multiple feature clusters are obtained by clustering the intermediate activation features. For each feature cluster, the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster is analyzed as the importance of that neuron. Finally, for each expert to be constructed, a predetermined number of neurons with high importance ranking are selected from the feedforward network layer according to the importance of the neurons and assigned to the neuron set corresponding to that expert; thus, a sparse hybrid expert model composed of multiple experts is obtained. The sparse hybrid expert model may include a shared expert and multiple specialized experts.

[0057] In an optional embodiment, step S150 above, "for the input token to be processed, select at least one target expert from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model," may include steps S150-1 to S150-4: Step S150-1: Input the input token into the shared expert and obtain the activation value of the input token in the shared expert as the token feature of the input token.

[0058] Specifically, when a new input token needs to be reasoned, it is first fed into a shared expert in a sparse hybrid expert model. The token flows through a unique set of neurons within the shared expert and undergoes corresponding computation. The activation value of a specific layer during this computation (e.g., an intermediate layer within the shared expert) is selected as the token feature of the input token. This token feature encodes the semantic representation of the token after processing with general knowledge, providing a unified benchmark for subsequent route matching.

[0059] Step S150-2: For each specialized expert, obtain the activation values ​​of the neurons assigned to that specialized expert when processing the calibration dataset, and determine the expert features of that specialized expert based on the activation values.

[0060] Specifically, during the model transformation phase (after step S140), forward propagation is performed again using the calibration dataset. For each specialist expert, their dedicated set of neurons is kept active, and the activation values ​​generated when processing calibration data belonging to their feature cluster or related parts are collected. These activation values ​​reflect the expert's response when encountering semantic patterns they are adept at processing. The collected activation values ​​are aggregated (e.g., their mean vector or principal components are computed) to generate a representative feature vector. This vector represents the specialist expert's expert features. This feature vector is saved for use in all subsequent inference processes.

[0061] Step S150-3: Calculate the degree of association between the token feature and each of the expert features. The degree of association is characterized by Euclidean distance or cosine similarity.

[0062] Euclidean distance is the linear distance between token features and expert features in space. The smaller the distance, the closer the two vectors are numerically, and the higher the correlation. Cosine similarity is the cosine of the angle between token features and expert features. The closer the cosine value is to 1, the more consistent the directions of the two vectors, and the higher the correlation. The calculated distance or similarity value is a quantitative expression of the correlation between the input token and each specialized expert.

[0063] Step S150-4: Select at least one target expert from the plurality of specialized experts based on the degree of correlation.

[0064] Specifically, all the correlation scores calculated in step S150-3 are sorted. If Euclidean distance is used, the M experts with the smallest distances (M is the preset number of activated experts, e.g., top-2) are selected, sorted from smallest to largest. If cosine similarity is used, the M experts with the largest similarities are selected, sorted from largest to smallest. These M selected experts are determined as the target experts for processing the input token. The routing decision is entirely based on the numerical calculation results of feature matching and does not involve any trainable parameters.

[0065] For example, using Euclidean distance to represent the degree of association between token features and expert features, the degree of association between token features and expert features, as well as the selected target expert index, can be expressed as:

[0066]

[0067] in, The token characteristics for the input token. For the expert characteristics of the i-th specialized expert, For Euclidean distance, This represents the degree of correlation between the token feature and the expert feature of the i-th specialized expert. The sequence numbers of the first m target experts selected; It is an exponentially normalized function. It is a function that finds the index corresponding to the maximum value.

[0068] The technical solution adopted in this embodiment focuses on feature matching and ranking in routing decision-making, eliminating the need for backpropagation or parameter updates. This fundamentally solves the problem of huge training overhead for routing networks in related methods. Furthermore, since expert features are pre-calculated static values, only a simple vector distance or similarity calculation is required during inference to complete the routing, introducing minimal additional computational overhead.

[0069] In an optional embodiment, step S160 above, "processing the input token using the at least one target expert to obtain the output result corresponding to the input token," may include steps S160-1 to S160-3: Step S160-1: Input the input token to the shared expert to obtain the output vector of the shared expert, and input the input token to the at least one target expert to obtain the output vector of each target expert.

[0070] Specifically, the shared expert performs feedforward computation (including linear transformation and nonlinear activation function) on the input token based on its own set of neurons, and outputs a shared expert output vector, which represents the model's basic transformation result on the input token based on general knowledge.

[0071] The input token is fed into each of the target experts selected in step S150. Each target expert independently performs feedforward computation on the input token based on its own unique set of neurons. After each target expert completes its computation, it outputs its target expert output vector for that input token. Each such vector represents the transformation result of the input token by the specialized expert from its unique professional knowledge perspective.

[0072] Step S160-2: Determine the aggregation weight corresponding to each target expert based on the degree of correlation between the token features of the input token and the expert features of each target expert.

[0073] The correlation degree calculated in step S150-3 can be reused as the basis for determining the weights. The correlation degree quantifies the semantic fit between the input token and each target expert. If the correlation degree uses cosine similarity (the value range is usually [-1, 1], with positive values ​​indicating similarity), the similarity value can be directly used as the aggregation weight, or a softmax function can be used to normalize the similarity of the target experts to obtain a set of aggregation weights that sum to 1. If the correlation degree uses Euclidean distance (the smaller the distance, the more relevant it is), the distance needs to be converted into a similarity measure first, for example, by taking its reciprocal or negative value, and then normalized.

[0074] Finally, an aggregation weight is assigned to each target expert. Additionally, a base aggregation weight can be assigned to the shared experts. This base weight can be a preset constant (e.g., 0.1), dynamically set according to task requirements, or determined based on a certain correlation between the shared expert's output and the token's features. The sum of the aggregation weights of all participating experts (shared experts and target experts) should be 1.

[0075] Step S160-3: Based on the aggregation weight, perform weighted fusion on the output vector of the at least one target expert and the output vector of the shared expert to obtain the output result corresponding to the input token.

[0076] Specifically, the output vector of the shared expert is multiplied by its aggregate weight, the output vector of each target expert is multiplied by its set weight, and then all weighted vectors are summed to obtain the output result corresponding to the input token.

[0077] The technical solution of this embodiment achieves synergy and balance between general knowledge and expert knowledge by feeding input tokens in parallel to shared experts and dynamically selected target experts, and then performing weighted fusion based on matching degree. The shared expert, acting as a general processor, provides a stable foundation for the final result, preventing severe degradation of output quality even in extreme cases where routing decisions are inaccurate, thus enhancing the model's robustness. The target expert's output, based on its high matching degree with the input token, provides targeted and specialized knowledge supplementation. Through dynamic weight allocation based on matching degree, the model can adaptively adjust the contribution ratio of general knowledge and specialized knowledge in the final output, thereby obtaining more accurate and smoother results when processing inputs with multiple semantics or ambiguous boundaries.

[0078] like Figure 3 As shown, Figure 3This is a schematic diagram of a representation-based parameterless routing method provided in this application embodiment. Through the parameterless routing method in steps S150 to S160, dynamic routing of tokens can be achieved without an additional learnable routing network. Specifically, for an input token to be processed, the input token is input to a shared expert, and the activation value of the input token in the shared expert is obtained as the token feature of the input token. For each specialized expert, the activation value of the neurons assigned to that specialized expert when processing the calibration dataset is obtained, and the expert feature of that specialized expert is determined based on the activation value. The correlation degree between the token feature and each expert feature is then calculated. Based on the magnitude of the correlation degree, at least one target expert is selected from the plurality of specialized experts. Finally, the input token is processed using the at least one target expert to obtain the output result corresponding to the input token.

[0079] Through the above embodiments, this solution utilizes a calibration dataset from the target application scenario to drive feature analysis and expert segmentation. It can automatically and losslessly convert dense language models into sparse hybrid expert models without relying on large-scale training, avoiding the additional training overhead and parameter burden of introducing learnable routing networks, and reducing the cost of model compression and adaptation. Expert segmentation based on neuron importance gives the model a structurally ordered knowledge structure. Combined with a parameterless feature matching routing mechanism, it can more accurately assign appropriate experts to tokens with different semantics, thereby improving the model's processing power and output quality. Simultaneously, static expert features and parameterless routing give the model good cross-scenario adaptability and stability during deployment. A high-performance sparse model can be obtained with only one target-oriented calibration and transformation, making it highly practical.

[0080] This application also provides a device for constructing a token-aware expert hybrid model based on a pre-trained model, referring to... Figure 4 As shown, Figure 4 This is a schematic diagram of a token-aware expert hybrid model construction device based on a pre-trained model provided in an embodiment of this application. The device includes: The acquisition module 410 is used to acquire a calibration dataset, wherein the sample data of the calibration dataset is selected from at least one of the following application scenarios: intelligent question answering, dialogue interaction, text summarization, and language translation text; Extraction module 420 is used to input the calibration dataset into a pre-trained dense language model and extract the intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset. The determination module 430 is used to determine the importance of each neuron in the feedforward network layer based on the intermediate activation features. The partitioning module 440 is used to partition the neurons in the feedforward network layer into multiple non-overlapping sets of neurons according to the importance of the neurons, and each set of neurons constitutes an expert, thus obtaining a sparse hybrid expert model. The selection module 450 is used to select at least one target expert from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model, for the input token to be processed; wherein the expert features are characterized based on the response of the subset of neurons constituting the expert to the calibration dataset; The processing module 460 is used to process the input token using the at least one target expert to obtain the output result corresponding to the input token.

[0081] In an optional embodiment, the determining module 430 is further configured to: The intermediate activation features are clustered to obtain multiple feature clusters; wherein the number of feature clusters is set to a preset number of experts, and each feature cluster corresponds to one expert; For each feature cluster, the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster is analyzed to determine the importance of that neuron.

[0082] In an optional embodiment, the determining module 430 is further configured to: calculate the norm of the weight vector corresponding to the neuron as the contribution; or, calculate the statistics of the activation value of the neuron when processing intermediate activation features belonging to the feature cluster as the contribution.

[0083] In an optional embodiment, the partitioning module 440 is further configured to: For each expert to be constructed, based on the importance of the neurons, a predetermined number of neurons with the highest importance ranking are selected from the feedforward network layer and assigned to the neuron set corresponding to that expert. The sparse hybrid expert model includes a shared expert and multiple specialized experts. The shared expert processes all input tokens, while the specialized experts process a subset of the input tokens.

[0084] In an optional embodiment, the selection module 450 is further configured to: The input token is input to the shared expert, and the activation value of the input token in the shared expert is obtained as the token feature of the input token; For each specialized expert, obtain the activation values ​​of the neurons assigned to that specialized expert when processing the calibration dataset, and determine the expert features of that specialized expert based on the activation values; Calculate the degree of association between the token feature and each of the expert features; Based on the degree of correlation, at least one target expert is selected from the plurality of specialized experts.

[0085] In an alternative embodiment, the degree of association is characterized by Euclidean distance or cosine similarity.

[0086] In an optional embodiment, the processing module 460 is further configured to: The input token is input to the at least one target expert to obtain the output vector corresponding to each target expert; Based on the matching degree between the token features of the input token and the expert features of each target expert, the aggregation weight corresponding to each target expert is determined; Based on the aggregate weights, the output vectors of all target experts are weighted and summed to obtain the output result corresponding to the input token.

[0087] It is understood that the token-aware expert hybrid model construction device based on the pre-trained model in the embodiments of this application can implement the token-aware expert hybrid model construction method based on the pre-trained model in the above embodiments. The token-aware expert hybrid model construction device based on the pre-trained model has the same advantages as the token-aware expert hybrid model construction method based on the pre-trained model compared to the prior art, and will not be repeated here.

[0088] This application also provides an electronic device, which is described in reference to... Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus for communication. The memory 510 stores a computer program that can run on the processor 520 to implement the steps of the token-aware expert hybrid model construction method based on a pre-trained model described in the embodiments of this application.

[0089] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model described in this application.

[0090] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model described in this application.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0092] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0096] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0097] The foregoing provides a detailed description of the method, apparatus, device, and medium for constructing a token-aware expert hybrid model based on a pre-trained model. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing a token-aware expert hybrid model based on a pre-trained model, characterized in that, include: Obtain a calibration dataset, wherein the sample data of the calibration dataset is selected from at least one of the following application scenarios: intelligent question answering, dialogue interaction, text summarization, and language translation text; The calibration dataset is input into a pre-trained dense language model, and intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset are extracted. Based on the intermediate activation features, the importance of each neuron in the feedforward network layer is determined; Based on the importance of neurons, the neurons in the feedforward network layer are divided into multiple non-overlapping sets of neurons, and each set of neurons constitutes an expert, resulting in a sparse hybrid expert model. For an input token to be processed, at least one target expert is selected from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model; wherein the expert features are characterized based on the response of the subset of neurons constituting the expert to the calibration dataset; The input token is processed using at least one target expert to obtain the output result corresponding to the input token.

2. The method for constructing a token-aware expert hybrid model based on a pre-trained model according to claim 1, characterized in that, Based on the aforementioned intermediate activation features, the importance of each neuron in the feedforward network layer is determined, including: The intermediate activation features are clustered to obtain multiple feature clusters; wherein the number of feature clusters is set to a preset number of experts, and each feature cluster corresponds to one expert; For each feature cluster, the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster is analyzed to determine the importance of that neuron.

3. The method for constructing a token-aware expert hybrid model based on a pre-trained model according to claim 2, characterized in that, For each feature cluster, the contribution of each neuron in the feedforward network layer to the intermediate activation features belonging to that feature cluster is analyzed, including: Calculate the norm of the weight vector corresponding to the neuron as the contribution; or, The contribution is calculated as the statistical measure of the activation value of the neuron when processing intermediate activation features belonging to the feature cluster.

4. The method for constructing a token-aware expert hybrid model based on a pre-trained model according to claim 1, characterized in that, Based on the importance of neurons, the neurons in the feedforward network layer are divided into multiple non-overlapping sets of neurons, including: For each expert to be constructed, based on the importance of the neurons, a predetermined number of neurons with the highest importance ranking are selected from the feedforward network layer and assigned to the neuron set corresponding to that expert. The sparse hybrid expert model includes a shared expert and multiple specialized experts. The shared expert processes all input tokens, while the specialized experts process a subset of the input tokens.

5. The method for constructing a token-aware expert hybrid model based on a pre-trained model according to claim 4, characterized in that, For the input token to be processed, based on the correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model, at least one target expert is selected from multiple experts, including: The input token is input to the shared expert, and the activation value of the input token in the shared expert is obtained as the token feature of the input token; For each specialized expert, obtain the activation values ​​of the neurons assigned to that specialized expert when processing the calibration dataset, and determine the expert features of that specialized expert based on the activation values; Calculate the degree of association between the token feature and each of the expert features; Based on the degree of correlation, at least one target expert is selected from the plurality of specialized experts.

6. The method for constructing a token-aware expert hybrid model based on a pre-trained model according to claim 5, characterized in that, The degree of correlation is characterized by Euclidean distance or cosine similarity.

7. The method for constructing a token-aware expert hybrid model based on a pre-trained model according to claim 5, characterized in that, The input token is processed using at least one target expert to obtain the output result corresponding to the input token, including: The input token is input to the shared expert to obtain the output vector of the shared expert, and the input token is input to the at least one target expert to obtain the output vector of each target expert; The aggregation weight corresponding to each target expert is determined based on the degree of correlation between the token features of the input token and the expert features of each target expert. Based on the aggregation weights, the output vectors of the at least one target expert and the output vectors of the shared expert are weighted and fused to obtain the output result corresponding to the input token.

8. A device for constructing a token-aware expert hybrid model based on a pre-trained model, characterized in that, include: The acquisition module is used to acquire a calibration dataset, wherein the sample data of the calibration dataset is selected from at least one of the following application scenarios: intelligent question answering, dialogue interaction, text summarization, and language translation text; An extraction module is used to input the calibration dataset into a pre-trained dense language model and extract intermediate activation features of at least one feedforward network layer of the dense language model for multiple tokens in the calibration dataset. A determination module is used to determine the importance of each neuron in the feedforward network layer based on the intermediate activation features; The partitioning module is used to divide the neurons in the feedforward network layer into multiple non-overlapping sets of neurons according to the importance of the neurons, and each set of neurons constitutes an expert, thus obtaining a sparse hybrid expert model. A selection module is used to select at least one target expert from multiple experts based on the degree of correlation between the token features of the input token and the expert features of each expert in the sparse hybrid expert model, for the input token to be processed; wherein the expert features are characterized based on the response of the subset of neurons constituting the expert to the calibration dataset; The processing module is used to process the input token using the at least one target expert to obtain the output result corresponding to the input token.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the token-aware expert hybrid model construction method based on a pre-trained model as described in any one of claims 1-7.

Images