Reasoning methods, devices, electronic devices, media, and programs for hybrid expert models

Abstract

This disclosure provides a reasoning method, apparatus, electronic device, medium, and program for a hybrid expert model. By acquiring the data to be reasoned using a hybrid expert model, the data is processed sequentially through multiple expert layers within the model. The reasoning result is output after processing by the last expert layer. Within each expert layer, for each unit, multiple candidate experts are selected from the multiple experts in that layer based on the original weights of each expert for that unit. Based on the static enhancement capabilities and original weights of these candidate experts, the expected contribution strength of each candidate expert for that unit is determined. Based on the expected contribution strength of each candidate expert for that unit, the target expert corresponding to that unit is selected from the multiple candidate experts. This improves the accuracy of reasoning and avoids the additional latency and memory bandwidth overhead caused by low-contribution experts.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a reasoning method, apparatus, electronic device, medium, and program using a hybrid expert model. Background Technology

[0002] As large language models and multi-task generation models continue to expand their parameter scale, MoE (Mixture of Experts) models, due to their characteristics of "large total number of parameters and few experts activated per token," have become an important path to improve model capacity and maintain deployability. A typical MoE model activates only a small number of experts on each token, thus theoretically resulting in a significant reduction in computational cost compared to fully dense feedforward networks.

[0003] In related technologies, the matching degree between tokens and experts is calculated using a gating network in the MoE model. The top-k experts with the highest matching degree are activated to process the token, and then the processing results of the top-k experts are linearly mixed. However, different experts have different weight structures, and relying solely on the matching degree between tokens and experts cannot reflect the experts' own abilities. This leads to inaccurate expert selection, with some experts with low abilities still being activated, resulting in poor model inference accuracy and causing additional latency and memory bandwidth overhead. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. This disclosure provides a reasoning method, apparatus, electronic device, medium, and program for hybrid expert models.

[0005] On one hand, embodiments of this disclosure provide a reasoning method using a hybrid expert model, including: Obtain the reasoning data to be processed by the hybrid expert model. The reasoning data is text or multimedia data to be processed. The reasoning data includes multiple units, each unit being a word element or a multimedia block. The hybrid expert model includes multiple expert layers, and each expert layer includes multiple experts. The data to be reasoned is processed sequentially through the multiple expert layers, and the reasoning result of the data to be reasoned is output after the last expert layer processes the data to be reasoned. The processing of the data to be reasoned through each expert level includes: For each of the plurality of units, based on the original weights of each expert in each expert layer for each unit, multiple candidate experts are selected from the multiple experts in each expert layer. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts to each unit is determined. Based on the expected contribution intensity of the multiple candidate experts to each unit, the target expert corresponding to each unit is selected from the multiple candidate experts; Each unit is processed by the target expert corresponding to each unit to obtain the processing result of each unit. The reasoning result of the data to be reasoned includes the processing results of the multiple units.

[0006] In another aspect, embodiments of this disclosure provide a hybrid expert model inference apparatus, comprising: The acquisition module is used to acquire the data to be reasoned by the hybrid expert model. The data to be reasoned is text or multimedia data to be processed. The data to be reasoned includes multiple units, each unit being a word or a multimedia block. The hybrid expert model includes multiple expert layers, and each expert layer includes multiple experts. The reasoning module is used to process the data to be reasoned through the multiple expert layers in sequence, and output the reasoning result of the data to be reasoned after the last expert layer has processed the data to be reasoned. Specifically, when the reasoning module processes the data to be reasoned through each expert layer, it is used for: For each of the plurality of units, based on the original weights of each expert in each expert layer for each unit, multiple candidate experts are selected from the multiple experts in each expert layer. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts to each unit is determined. Based on the expected contribution intensity of the multiple candidate experts to each unit, the target expert corresponding to each unit is selected from the multiple candidate experts; Each unit is processed by the target expert corresponding to each unit to obtain the processing result of each unit. The reasoning result of the data to be reasoned includes the processing results of the multiple units.

[0007] In one possible implementation, the apparatus further includes a determining module, which, when determining the static enhancement capabilities of multiple experts in each expert layer, is specifically used for: Determine the diagonal matrix of the normalized scaling vector for each expert layer; For each expert in each expert layer, the original static enhancement capability of each expert is determined based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert. The original static enhancement capabilities of multiple experts in each expert layer are normalized to obtain the static enhancement capabilities of the multiple experts.

[0008] In one possible implementation, the determining module, when determining the original static enhancement capability of each expert based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert, is specifically used for: Based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert, the static enhancement capability of the activation branch of each expert and the static enhancement capability of the gated branch of each expert are determined respectively. The static enhancement capabilities of the activated branches and the gated branches of each expert are averaged to obtain the original static enhancement capability of each expert.

[0009] In one possible implementation, the inference module, when selecting the target expert corresponding to each unit from the multiple candidate experts based on the expected contribution intensity of each candidate expert to each unit, is specifically used for: The expected contribution intensity of the multiple candidate experts to each unit is normalized to obtain the expected contribution score of the multiple candidate experts to each unit. Based on the contribution intensity threshold of each expert layer and the expected contribution scores of the plurality of candidate experts, a mask is determined for each candidate expert among the plurality of candidate experts, so as to retain the target expert among the plurality of candidate experts and prune the non-target experts among the plurality of candidate experts except the target expert; wherein, the mask of each candidate expert indicates whether to retain each candidate expert.

[0010] In one possible implementation, when the inference module processes each unit based on the target expert corresponding to each unit to obtain the processing result of each unit, it is specifically used for: Based on the mask of each candidate expert, the original weights of the multiple candidate experts are normalized to obtain the pruning weights of the multiple candidate experts. Each of the multiple candidate experts processes each unit separately, thereby obtaining the processing results of the multiple candidate experts for each unit; Based on the pruning weights of the multiple candidate experts, the processing results of the multiple candidate experts on each unit are weighted to obtain the processing results of each expert layer on each unit after pruning.

[0011] In one possible implementation, the apparatus further includes a threshold acquisition module, which, when acquiring the contribution intensity threshold for each expert layer, is specifically used for: Obtain multiple candidate thresholds; For each of the plurality of candidate thresholds, each expert layer is pruned based on the calibration text and each candidate threshold to determine each calibration candidate expert before pruning and the calibration target expert after pruning. Based on each calibration candidate expert before pruning and the calibration target expert after pruning, the mean squared error and pruning rate corresponding to each candidate threshold are determined; wherein, the mean squared error represents the difference between the processing result of the calibration text before pruning and the processing result of the calibration text after pruning based on each candidate threshold; the pruning rate represents the ratio of experts in each expert layer to be pruned based on each candidate threshold; From the plurality of candidate thresholds, a plurality of first candidate thresholds whose mean squared error satisfies the error budget condition are selected, and from the plurality of first candidate thresholds, a target candidate threshold corresponding to the maximum pruning rate is selected as the contribution intensity threshold of each expert layer.

[0012] In one possible implementation, when the threshold determination module prunes each expert layer based on the calibration text and each candidate threshold, and determines each calibration candidate expert before pruning and the calibration target expert after pruning for each expert layer, it is specifically used for: Based on the original weights of each expert in each expert layer for the calibration text, multiple calibration candidate experts are selected from multiple experts. Based on the static enhancement capabilities and original weights of the multiple calibration candidate experts, the expected contribution strength of the multiple calibration candidate experts to the calibration text is determined. For each of the plurality of candidate thresholds, a mask for each calibration candidate expert is determined based on each candidate threshold and the expected contribution score of the plurality of calibration candidate experts, so as to obtain the pruned calibration target expert in each expert layer based on the mask of the calibration candidate expert.

[0013] In one possible implementation, when the threshold determination module determines the mean square error and pruning rate corresponding to each candidate threshold based on each calibration candidate expert before pruning and the calibration target expert after pruning, it is specifically used for: The calibration text is processed based on multiple calibration candidate experts to obtain a first processing result for each expert layer on the calibration text, and the calibration text is processed based on the calibration target expert to obtain a second processing result for each expert layer on the calibration text. Based on the first processing result and the second processing result, the mean square error corresponding to each candidate threshold is determined. Based on the masks of the multiple calibration candidate experts, the pruning rate corresponding to each candidate threshold is determined.

[0014] In another aspect, embodiments of this disclosure provide an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the above-described hybrid expert model reasoning method.

[0015] In another aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the aforementioned hybrid expert model reasoning method.

[0016] In another aspect, embodiments of this disclosure provide a computer program, including computer program instructions, which, when executed by a processor, implement the above-described hybrid expert model reasoning method.

[0017] The technical solution provided by this invention obtains the data to be reasoned using a hybrid expert model, processes the data sequentially through multiple expert layers in the hybrid expert model, and outputs the reasoning result of the data after processing by the last expert layer. Specifically, within each expert layer, for each unit, multiple candidate experts are selected from the multiple experts in that expert layer based on the original weights of each expert for that unit. Based on the static enhancement capabilities and original weights of these multiple candidate experts, the expected contribution intensity of each candidate expert for that unit is determined. Based on the expected contribution intensity of these multiple candidate experts for that unit, the target expert corresponding to that unit is selected from the multiple candidate experts. Based on this, the contributions and original weights of the candidate experts can be combined to accurately activate effective contributing experts with high static enhancement capabilities among the candidate experts. The unit is then processed based on the target expert to obtain the processing result, ensuring that each unit is accurately processed by experts with actual effective contributions, thereby improving the accuracy of reasoning and avoiding the additional latency and memory bandwidth overhead caused by low-contribution experts.

[0018] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart of a reasoning method using a hybrid expert model provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating another inference method using a hybrid expert model provided in this embodiment of the disclosure; Figure 3 A schematic diagram illustrating a static enhancement capability push-out provided in an embodiment of this disclosure; Figure 4 A flowchart illustrating another inference method using a hybrid expert model provided in this embodiment of the disclosure; Figure 5 A schematic diagram of a token-level dynamic expert pruning process provided in an embodiment of this disclosure; Figure 6 A flowchart illustrating another inference method using a hybrid expert model provided in this embodiment of the disclosure; Figure 7 A flowchart illustrating another inference method using a hybrid expert model provided in this embodiment of the disclosure; Figure 8 This is a schematic diagram of the overall process of reasoning provided in an embodiment of the present disclosure; Figure 9 A block diagram of a hybrid expert model inference apparatus provided in an embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0021] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0022] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0023] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0024] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0025] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0026] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0027] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0028] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0029] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0030] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0031] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0032] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0033] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0034] In the current field of large-scale artificial intelligence models, in addition to the classic Transformer architecture, the MoE architecture has emerged in recent years. The MoE architecture is a machine learning architecture that combines multiple specialized sub-models (called "experts" or "expert modules") together. The MoE architecture divides the parameters in the model into multiple different "experts," each specializing in handling a specific type of input or task, thereby achieving more efficient knowledge utilization and task processing, becoming one of the key architectures driving the computational performance of large models. The MoE architecture uses a gating network to dynamically determine which experts should be used when processing each input, thus leveraging the strengths of multiple experts to handle complex tasks and improve the model's performance and generalization ability. The MoE architecture possesses strong expertise and reasoning capabilities in specific domains, and through the collaboration of experts from multiple different domains, it can solve complex and comprehensive problems.

[0035] The core idea of ​​the MoE model is to assign input data to different experts and then merge the outputs of all experts to generate the final result. This assignment can be dynamically adjusted based on the characteristics of the input data, ensuring that each expert handles the data type or task aspect they are best at, thereby achieving more efficient and accurate predictions. By using multiple expert networks to process different tasks or features, each token input into the model can activate different experts. The model can dynamically select the appropriate expert for processing based on the characteristics of the input, improving the model's flexibility and expressiveness, while reducing the model's computational cost and parameter size while ensuring performance.

[0036] The MoE model comprises multiple expert layers, each containing several experts, and each expert can be an independent neural network. Each expert layer includes a routing module G and M experts E = {E1, ..., E...}. MExperts can be independent neural networks that specialize in processing specific subsets of input data or specific tasks. The router, also known as a gating network, is an independent neural network whose main task is to dynamically determine which expert(s) should process each token based on the characteristics of the input data, and to assign appropriate weights to each expert. For a single token input to the model, only a few experts are activated for inference at each inference iteration; the gating unit determines which experts perform inference or which tokens are assigned to which experts. Specifically, at each layer of the model, the router network of the gating unit selects k (where k is an integer greater than or equal to 1) experts to process the current state and combine their outputs.

[0037] Figure 1 This is a flowchart illustrating an exemplary embodiment of the inference method using a hybrid expert model provided in this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps: Step 101: Obtain the data to be reasoned for the hybrid expert model. The data to be reasoned is text or multimedia data to be processed. The data to be reasoned includes multiple units, each unit being a word or a multimedia block. The hybrid expert model includes multiple expert layers, and each expert layer includes multiple experts.

[0038] The hybrid expert model can be a large language MoE model focused on NLP (Natural Language Processing) tasks, and the data to be inferred can be text, specifically user input text in tasks such as translation, long text understanding, and general dialogue text reasoning. For example, in a dialogue text reasoning scenario, if the data to be inferred is the question text input by the user into the large language MoE model, then each unit is a token in the question text.

[0039] Alternatively, the hybrid expert module can also be a Vision-MoE model, where the data to be inferred can be multimedia data such as images, audio, and video. Specifically, it can be multimedia data from large-scale image classification, object detection, and image segmentation tasks. For example, in an image classification scenario, the data to be inferred is a user-input image to be classified, and each unit can be a single image block within a larger set of image blocks.

[0040] Alternatively, the hybrid expert module can also be a multimodal MoE model, and the data to be inferred can be a mixture of text and multimedia data. Specifically, it can be multimodal data input by users in complex cross-modal task scenarios such as embodied intelligent robot control, image description generation, and visual question answering. For example, in a visual question answering scenario, the data to be inferred is user-input question text with an image. The image can be divided into image blocks, and the question text can be divided into word units. Each unit can be a word unit or an image block.

[0041] This hybrid expert model comprises multiple expert layers, each containing multiple experts. Each expert can be an independent feedforward neural network, such as an MLP (Multilayer Perceptron) network or a SwiGLU MLP (Swish-Gated Linear Unit) network.

[0042] In one possible example, for a certain expert layer, the input hidden state of a single token in the data to be inferred can be represented as a vector. ,in, ; The hidden size (hidden layer dimension) represents the length of the hidden state vector, which is d. The token is mapped to a d-dimensional feature space.

[0043] Taking the SwiGLU MLP network as an example, each expert includes an activation branch and a gated branch. The token features of the expert are input and processed by the activation branch and the gated branch respectively to obtain the expert's processing result of the input token features. Specifically, it includes the following steps 1 to 4: 1. In the activation branch, the input token features are linearly transformed by the weight matrix of the activation branch; The weight matrix of the activation branch can be an up-projection matrix, which can project the input token features to a higher-dimensional feature space.

[0044] For example, the activation branch weight matrix can be represented as ,in, , It is a d x m matrix that can represent the d-dimensional token feature vector. Mapped to a higher m-dimensional dimension. m represents the intermediate size of the expert.

[0045] 2. In the gated branch, the input token features are linearly transformed by the weight matrix of the gated branch, and then activated by the Swish activation function. The weight matrix of the gated branch can be an up-projection matrix; for example, the weight matrix of the gated branch can be expressed as: ,in, , It is a d x m matrix that can represent the d-dimensional token feature vector. The result is mapped to a higher m-dimensional dimension. The linear processing of the m-dimensional dimension is then further activated using the Swish activation function, for example, the sigmoid function.

[0046] 3. Using the gating mechanism of the Swish MLP network, the activation result processed by the Swish activation function is multiplied element-wise with the result processed by the activation branch weight matrix to obtain the gating result; In the gating mechanism, the activation function can be used to process the results and determine which dimensions of information in the processing results of the activated branch are retained or suppressed. Based on this, the information flow can be dynamically adjusted to filter out effective information.

[0047] 4. Perform a linear transformation on the gating result according to the expert's output projection matrix to obtain the expert's processing result of the input token features.

[0048] Here, the output projection matrix is ​​a dimension-reduced projection matrix, which can be expressed as: , ,pass The gating results are mapped back to the original d-dimensional vector to achieve dimensionality reduction and integrate high-dimensional features. The nonlinear semantic information learned by the experts is compressed into a vector with the same dimension as the input, which is convenient for subsequent modules to process.

[0049] Step 102: The data to be reasoned is processed sequentially through the multiple expert layers, and the reasoning result of the data to be reasoned is output after the last expert layer processes the data to be reasoned.

[0050] In this step, the data to be inferred can be input into the hybrid expert model and processed sequentially through multiple expert layers. The processing result of each expert layer is output to the next expert layer for processing, until the last expert layer processes the data and outputs the inference result.

[0051] The processing of the data to be reasoned through each expert level includes the following steps 1021 to 1024: Step 1021: For each of the multiple units, based on the original weight of each expert in each expert layer for each unit, select multiple candidate experts from the multiple experts in each expert layer.

[0052] Each expert layer consists of a gating network and multiple experts. Because the gating network has the dynamic routing decision-making function of dynamically allocating different tokens to the appropriate experts, it is also called a routing module.

[0053] The original weights represent the degree of matching between experts and units. The larger the original weight value, the higher the degree of matching between the expert and the unit, and the higher the probability that the expert will be selected as a candidate expert. For each expert layer, the matching degree is calculated between each unit in the inference data and each expert in that layer through the gating network of that expert layer, obtaining the gate score of each expert, which is also the original weight of each expert. For example, the gating network can perform a linear transformation on the input token features using a pre-trained weight matrix to map the token to the expert space, generating the original weight of each expert. Furthermore, based on the original weights of multiple experts, multiple candidate experts are selected from multiple experts; for example, the top-k (the first k, where k is an integer greater than 1) candidate experts with the highest original weights can be selected.

[0054] In one possible example, taking the unit as a token, for a certain token, the gating network in the th hybrid expert model... The top-k candidate experts selected by the layer screening are represented as follows: .in, Let represent the set of experts including the top (k) candidate experts. The gate score corresponding to the i-th candidate expert among the top (k) candidate experts can be represented as ,in, ; .in, For the first The number of each candidate expert. Let be the original weight assigned by the gating network to the i-th candidate expert for the current token. This indicates the characteristics of the current token. It represents the degree of matching between the current token and the i-th candidate expert, that is, whether the current token needs the i-th candidate expert.

[0055] Step 1022: Based on the static enhancement capabilities and original weights of the multiple candidate experts, determine the expected contribution intensity of the multiple candidate experts to each unit.

[0056] Static enhancement capability characterizes the expert's ability to amplify the static output of the input. Specifically, static enhancement capability characterizes the expert's ability to activate, amplify, transform, and write back the features of the input token. For example, by using activation branch weight matrices and gated branch weight matrices to perform dimensionality upscaling, linear transformation, and selective enhancement and nonlinear amplification of the input token features through gating mechanisms, activation amplification and transformation can be achieved, enhancing the feature representation capability of the gating results. For transformation capability, feature selection and nonlinear transformation can be performed using gating mechanisms. For example, by using the output projection matrix to reduce the dimensionality of the gating results and projecting the high-dimensional features back to the original input dimension, information can be written back.

[0057] For each expert, after the hybrid expert model is trained, the static augmentation capability of the expert can be evaluated in advance using the weight matrix of the activation branch, the weight matrix of the gated branch, and the output projection matrix, and the evaluated static augmentation capability of each expert can be stored.

[0058] In this step, the static enhancement capabilities of each pre-evaluated candidate expert can be directly obtained. Based on the static enhancement capabilities and original weights of each candidate expert, the expected contribution strength of each candidate expert to each input token can be comprehensively evaluated. The specific implementation method for evaluating the expected contribution strength based on static enhancement capabilities and original weights is detailed below, as shown in formula (14) below.

[0059] The expected contribution strength characterizes the expert's ability to contribute to each unit. Specifically, the expected contribution strength not only represents the degree of matching between each input token feature and the candidate expert, but also the candidate expert's ability to activate, amplify, transform, and write back the token feature, thus reflecting the candidate expert's true contribution to the token feature.

[0060] Step 1023: Based on the expected contribution intensity of the multiple candidate experts to each unit, select the target expert corresponding to each unit from the multiple candidate experts.

[0061] In this step, experts whose expected contribution intensity is not lower than the contribution intensity threshold can be selected from the top-k candidate experts as target experts. For example, 3 or 1 target experts can be selected from the top-8 candidate experts. Within each expert layer, the target expert corresponding to each unit in that layer is the expert who actually processes that unit, and the processing result of that unit in that expert layer is obtained through the processing of that unit by the target expert.

[0062] Step 1024: Based on the target expert corresponding to each unit, process each unit to obtain the processing result of each unit.

[0063] The reasoning result of the data to be reasoned includes the processing results of the multiple units. In this step, for each unit, the unit can be input into the target expert corresponding to that unit for processing to obtain the target expert's processing result for that unit. Specifically, for each unit, if the target expert corresponding to the unit includes one expert, then the processing result of that target expert is used as the processing result of the current expert layer for that unit; if the target expert corresponding to the unit includes two or more experts, then the processing results of each expert can be weighted according to the original weights of each expert included in the target expert to obtain the final processing result of the current expert layer for that unit.

[0064] Based on this, the processing results corresponding to multiple units in the data to be reasoned can be obtained. Then, the processing results corresponding to multiple units in the data to be reasoned can be input into the next expert layer. Steps 1021 to 1024 are repeated for the next expert layer to obtain the processing results of the next expert layer for the data to be reasoned, until the processing result of the last expert layer is obtained. Based on the processing result of the last expert layer, the final reasoning result of the data to be reasoned is obtained.

[0065] In one possible scenario, within a dialogue text reasoning context, the MoE model is a large language MoE model used for NLP tasks. The data to be reasoned about is the question text input by the user into the large language MoE model, and the reasoning result can be the answer text corresponding to the question text. For example, in the field of legal consultation, the large language MoE model reasons about the user's input request and outputs the corresponding legal answer.

[0066] In another possible scenario, in image classification, the MoE model is a visual hybrid expert model. The data to be inferred is the image to be classified input by the user, and the inference result can be the type of the image to be classified, output in text form. For example, in the fields of intelligent transportation and autonomous driving, the visual hybrid expert model identifies the categories of traffic signs in images of the surrounding environment captured by vehicle cameras, classifying speed limit signs, stop signs, yield signs, etc. in real time.

[0067] In another possible scenario, visual question answering, the MoE model is a multimodal MoE model. The data to be inferred is the user-input question text accompanied by an image, and the inference result can be in text form or multimodal data containing text, images, and other formats. For example, in the meteorological field, the multimodal MoE model infers from visual features (such as raindrops and snow spots) in images of the current weather taken by a camera, and outputs classification and analysis results of the current weather.

[0068] In related technologies, during MoE model inference, the gating network only activates the top-k experts of the MoE model on each token according to the original weights. Further research by the inventors of this invention has revealed the following technical problems with this approach: 1. Although the routing module only activates the top-k experts, not all of these experts contribute equally to the current token. 2. Existing technologies typically execute all the top-k experts selected by the routing module directly, and then linearly mix the processing results of the top-k experts according to their original weights. This results in some low-contribution experts still being fully computed, causing additional latency and memory bandwidth overhead. 3. Threshold pruning based solely on the original weights of the routing module often ignores the differences in the "structural amplification capabilities" of the experts themselves. Two experts with similar original weights may have significantly different actual output capabilities.

[0069] The inference method of the hybrid expert model provided in this invention obtains the data to be inferred from the hybrid expert model, processes the data sequentially through multiple expert layers in the hybrid expert model, and outputs the inference result of the data after processing by the last expert layer. Specifically, in each expert layer, for each unit, multiple candidate experts are selected from the multiple experts in that expert layer based on the original weights of each expert for that unit. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts for that unit is determined. Based on the expected contribution intensity of the multiple candidate experts for that unit, the target expert corresponding to that unit is selected from the multiple candidate experts. Based on this, the contributions and original weights of the candidate experts can be combined to accurately activate the effective contributing experts with high static enhancement capabilities among the candidate experts, and the unit is processed based on the target expert to obtain the processing result. This ensures that each unit is accurately processed by the experts who actually make effective contributions, thereby improving the accuracy of inference and avoiding the additional latency and memory bandwidth overhead caused by low-contribution experts.

[0070] In one possible embodiment, Figure 2 This is a flowchart illustrating another inference method using a hybrid expert model provided in an exemplary embodiment of this disclosure. Figure 2 As shown, for each expert layer, the static enhancement capabilities of multiple experts in each expert layer are determined by performing the following steps A1-A3: Step A1: Determine the diagonal matrix of the normalized scaling vector for each expert layer; In the expert layer employing a feedforward network structure, an RMSNorm (Root Mean Square Layer Normalization) layer may be included. The input to the expert's SwiGLU MLP network is then processed after passing through the RMSNorm layer. The RMSNorm layer is used to normalize the input token features.

[0071] The RMSNorm layer may include a normalized scaling vector. , In this step, the diagonal matrix can be constructed based on the normalized scaling vector.

[0072] For example, normalized scaling vectors It is a d-dimensional vector that can be used to... Each element is arranged along the main diagonal, and the remaining positions are filled with zeros, resulting in a diagonal matrix. , The specific matrix form is as follows: .

[0073] Step A2: For each expert in each expert layer, determine the original static augmentation capability of each expert based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert. In this step, for each expert, the expert's ability to enhance the original static features of the input token can be evaluated based on the weight matrix and projection matrix that have been trained in the expert's internal network structure.

[0074] In one possible embodiment, the static enhancement capabilities of different branches can be determined separately based on the weight matrices of the activation branches and the gating matrix, and the static enhancement capabilities of the experts can be further evaluated by combining the capabilities of the different branches. Specifically, the execution process of step A2 may include the following steps: determining the static enhancement capabilities of the activation branches and the gating branches of each expert based on the weight matrices of the activation branches, the weight matrices of the gating branches, the output projection matrix, and the diagonal matrix of each expert; averaging the static enhancement capabilities of the activation branches and the gating branches of each expert to obtain the original static enhancement capability of each expert.

[0075] Among them, for the first An expert is defined by three sets of matrices, including: activation branch weight matrix. Gated branch weight matrix Output projection matrix , .

[0076] In this step, for each expert, the static enhancement capability of the active branch perspective can be calculated using the following formula (1): Formula (1) Furthermore, the static enhancement capability of the expert's gated branch perspective is calculated using the following formula (2): Formula (2) And, the expert's original static enhancement capability is calculated using the following formula (3): Formula (3) in, This indicates the static enhancement capability of activating the branch perspective. This indicates the static enhancement capability of the gated branch view. Indicates the original static enhancement capability; It is a numerically stable term.

[0077] It should be noted that, Figure 3 This is a schematic diagram illustrating a static enhancement capability push provided by the present invention. Figure 3 This demonstrates the derivation relationships from standard MLP, SwiGLU MLP to a single expert in MoE, and... , , The construction process. For example... Figure 3 As shown, the inventors of this invention have discovered that, for a standard pre-norm MLP (feedforward multilayer perceptron) module, its output, ignoring bias, can be expressed as the following formula (4): Formula (4) in: This represents the activation function, which is typically SiLU or a similar smooth monotonic activation function in some hybrid expert models. This represents the normalized residual input. This is the output of the MLP branch.

[0078] If we approximate that the input norm satisfies the following after normalization: Furthermore, it is assumed that the activation function will not significantly amplify the norm, i.e.: ,like Figure 3 As shown, based on formula (4), we can obtain an approximation of the following formula (5): Formula (5) Based on this, the Frobenius norm can be used to measure the overall static amplification capability of the MLP module for the hidden state, such as... Figure 3 As shown, it is specifically represented by the following formula (6): Formula (6) in, Denotes the Frobenius norm. This indicates the static amplification capability of the MLP module. The larger the value, the stronger the overall activation amplification capability of the MLP module for the input, provided that the input is normalized.

[0079] After further research by the inventors of this invention, such as Figure 3 As shown, extending from the MLP module to the SwiGLU structure, the output of the SwiGLU module can be expressed as the following formula (7): Formula (7) in: This represents element-wise multiplication. This is the output of the active branch of the SwiGLU module. This is the gated branch output of the SwiGLU module.

[0080] For formula (7), let , Then the branch output in the SwiGLU module after gating can be expressed as In static scaling estimation, this output can be approximated as the combined effect of "activated branch amplitude" and "gated branch propagation path," such as... Figure 3 As shown, it is specifically represented by the following formula (8): Formula (8) Given that the input has been normalized, we can further approximate the static enhancement capability of the SwiGLU module, such as... Figure 3 As shown, it is specifically represented by the following formula (9): Formula (9) in, An approximate representation of the static enhancement capabilities of the SwiGLU module. Reflects the static amplification intensity of the active branch in the SwiGLU module. This reflects the linear propagation capability from the gated branch to the output write-back in the SwiGLU module. This represents the identity term corresponding to the shared residual path.

[0081] Further research by the inventors of this invention revealed that, when extended to the MoE model, each expert in the MoE model essentially corresponds to an independent SwiGLU feedforward module. However, during hybridization, typically only the feedforward output branch of the expert is considered, and the shared residual is not included in the individual expert's capability. Therefore, for experts... Its feedforward output is defined as follows (10): Formula (10) in, This is the expert-only output, excluding shared residual terms. In other words, it represents the feedforward output calculated independently by each expert and does not include shared residual connections. It is calculated by the expert's SwiGLU feedforward network, meaning only the expert-only FFN (Feed-Forward Network) branches are retained. This reflects the expert's own transformation capabilities and can be used to distinguish individual skill differences between experts. Based on this, the goal of retaining only the parts that differentiate the individual abilities of different experts can be achieved.

[0082] Based on the formula (10) obtained by the inventors of this invention through the above research and reasoning, as follows: Figure 3 As shown, further reasoning constructs formulas (1) to (2) above, which are the static enhancement capabilities corresponding to the activated branch perspective and the gated branch perspective, respectively. The two perspectives are then geometrically averaged and fused using formula (3) to calculate the original static enhancement capability of each expert. The static enhancement capability of each expert is then obtained by in-layer normalization using formula (11) below. .

[0083] Step A3: Normalize the original static enhancement capabilities of multiple experts in each expert layer to obtain the static enhancement capabilities of those multiple experts.

[0084] In this step, for each expert layer, the following formula (11) can be used to perform intra-layer normalization on all experts within that expert layer to obtain the final static enhancement capability of each expert: Formula (11) in, This represents the static enhancement capability of the e-th expert. This represents the original static enhancement capability of the e-th expert. Indicates the first Total number of experts at each level.

[0085] In this step, the static enhancement capabilities of each expert are calculated through steps A1 to A3 above. Then, multiple experts in each expert level can be... The values ​​are stored.

[0086] Based on this embodiment, by evaluating the static enhancement capabilities of the activation branches and gating branches in the expert from their respective perspectives, the static enhancement capabilities of the entire expert are further comprehensively measured by integrating different branches. This allows the static enhancement capability to characterize the activation amplification, transformation, and write-back capabilities of the entire network within the expert, from input activation and gating mechanisms to output, thereby improving the accuracy of the expert capability assessment.

[0087] In one possible embodiment, Figure 4 This is a flowchart illustrating another inference method using a hybrid expert model provided in an exemplary embodiment of this disclosure. Figure 4 As shown, step 1023 can be achieved through the following steps B1 to B2: Step B1: Normalize the expected contribution intensity of the multiple candidate experts for each unit to obtain the expected contribution score of the multiple candidate experts for each unit. In this step, for each unit, the expected contribution intensity of the top-k candidate experts corresponding to that unit can be normalized using the following formula (12) to obtain the expected contribution score of each candidate expert to that unit:

[0088] in, Let represent the expected contribution score of the i-th candidate expert among k candidate experts. This represents the expected contribution intensity of the i-th candidate expert. It is a numerically stable term.

[0089] Step B2: Based on the contribution intensity threshold of each expert layer, determine the mask of each candidate expert among the multiple candidate experts, so as to retain the target expert among the multiple candidate experts and prune the non-target experts among the multiple candidate experts except the target expert, wherein the mask of each candidate expert indicates whether to retain each candidate expert.

[0090] The mask for the target expert indicates that the target expert should be retained, while the mask for the non-target expert indicates that the non-target expert should be clipped.

[0091] Among them, for the first l For the expert layer, the mask for each candidate expert in this layer can be determined using the following formula (13): Formula (13) in, Indicates an indicator function, Indicates the first l The contribution strength threshold corresponding to the expert layer. This represents the expected contribution score of the i-th candidate expert.

[0092] in, This represents the mask for the i-th candidate expert. In formula (13), when the condition is met, that is, when... hour, If yes, the candidate expert will be retained; otherwise... This means skipping the candidate expert, or cutting off the candidate expert when the condition is not met.

[0093] In one possible example, to avoid the extreme case where all experts are eliminated, this invention adds a safety net constraint: regardless of changes in the contribution strength threshold, at least the expert with the highest expected contribution score is retained; that is, And force to satisfy: Based on this, by pre-configuring the mask of the expert with the highest expected contribution score to 1, the expert with the highest expected contribution score will not be pruned even if it is below the contribution intensity threshold.

[0094] Figure 5 This is a schematic diagram of a token-level dynamic expert pruning process provided by the present invention. It should be noted that for each unit, such as... Figure 5 As shown, for example, the hidden state of the input token. x (Hidden layer features) x ), the routing module in the The expert layer, through forward computation, uses the original weights to select a set of the top-k candidate experts. Specifically, it can be expressed as: Furthermore, the original weights of the top-k candidate experts can be represented as gate. The original weights of the i-th candidate expert Specifically, it can be expressed as: .in, The th among the top-k candidate experts The number of each candidate expert. This can represent the original weights that the routing module assigns to the i-th candidate expert based on the features of the current input token.

[0095] like Figure 5 As shown, further, based on the expert layer ID and the ID of each expert, the `amp_table[layer_id]` (mixed precision configuration table [indexed by expert layer ID]) and `[expert_id]` (expert ID index) are queried to obtain the information of each expert in that expert layer. e Static enhancement capability This includes pre-determining the static enhancement capabilities of each expert in each expert layer according to their respective expert layer IDs and the individual expert IDs within each expert layer. Store in amp_table[layer_id].

[0096] In step 1022, for each candidate expert, the expected contribution strength of each expert can be determined by combining the original weights of the routing module with the expert's static enhancement capabilities. Specifically, this can be based on the static enhancement capabilities and original weights of each candidate expert, such as... Figure 5 As shown, the expected contribution intensity of each candidate expert to each unit is calculated using the following formula (14): Formula (14) in, The weight represents the original weight of the i-th candidate expert, which can characterize whether the current input token feature needs this candidate expert. This represents the static output amplification capability of the i-th candidate expert. It represents the expected contribution intensity of the i-th candidate expert, which characterizes the assessment of the degree of contribution of the i-th candidate expert among the top-k candidate experts of each unit to that unit.

[0097] Furthermore, such as Figure 5 As shown, the expected contribution score of each candidate expert to the unit is calculated by normalizing using the above formula (12). Then, threshold screening is performed using formula (13) to determine the mask for each candidate expert, and a minimum retention rule is used to force their retention. .

[0098] It should be noted that the expert's true contribution norm can be approximately expressed by the following formula (15): Formula (15) in, This represents the processing result output by the i-th candidate expert after processing the input related items. "This can be the input token features. When focusing only on ranking and pruning the top-k candidate experts using prior knowledge such as weight matrices and projection matrices, relevant input terms can be treated as terms of the same order, thus..." As a calculable expected contribution strength that can be obtained before forward execution.

[0099] Based on this embodiment, by normalizing the expected contribution intensity of multiple candidate experts, expected contribution scores are obtained for each candidate expert. These scores, combined with the expert's static amplification ability, measure the expert's importance to the current token. The top-k candidate experts are dynamically pruned based on their importance scores, retaining only those with high contributions to the current token. Therefore, the normalized expected contribution scores can be used to filter out target experts with actual effective contributions, ensuring accurate evaluation of expert contributions, improving the precision of expert contribution assessment, and consequently enhancing the accuracy of inference.

[0100] In one possible embodiment, Figure 6 This is a flowchart illustrating another inference method using a hybrid expert model provided in an exemplary embodiment of this disclosure. Figure 6 As shown, step 1024 can be achieved through the following steps C1 to C3: Step C1: Based on the mask of each candidate expert, normalize the original weights of the multiple candidate experts to obtain the pruning weights of the multiple candidate experts. In this step, for the retained target experts, the original weights can be renormalized on a subset of the top-k candidate experts without directly changing the original semantics of the routing module.

[0101] For example, such as Figure 5 As shown, the original weight gate can be renormalized using the following formula (16) to calculate the pruning weight of each candidate expert among the top-k candidate experts:

[0102] in, This represents the pruning weight of the i-th candidate expert. This represents the mask for the i-th candidate expert. This represents the original weight of the i-th candidate expert. The expected contribution score of the i-th candidate expert... hour, , Not zero; otherwise This indicates that the candidate expert has been eliminated. It is a numerically stable term.

[0103] Step C2: Each of the multiple candidate experts processes each unit separately to obtain the processing results of the multiple candidate experts for each unit; For the i-th candidate expert The candidate expert has specific features regarding the input token. The processing result can be expressed as .

[0104] In one possible example, for each unit, the expected contribution score among the top-k candidate experts. Candidate experts, i.e., those with a mask of 0, can be directly defined as having a processing result of 0 for this unit. The expected contribution score among the top-k candidate experts is... The target expert, that is, the target expert with a mask of 1, can be used to actually execute the processing of the unit and obtain the processing result.

[0105] Step C3: Based on the pruning weights of the multiple candidate experts, the processing results of the multiple candidate experts for each unit are weighted to obtain the processing results of each expert layer for each unit after pruning.

[0106] In this step, such as Figure 5 As shown, the pruned processing results can be obtained by weighting the processing results of the top-k candidate experts based on the pruning weights of the top-k candidate experts using the following formula (17):

[0107] in, This indicates the processing result of the expert layer for each unit; for example, the first... l The expert layer focuses on the features of a single token in the data to be inferred. x The processing results.

[0108] Based on this embodiment, by using a mask to calculate the pruning weight of each candidate expert, and further using the pruning weight to weight the processing results of the top-k candidate experts, only the participants in the calculation are changed, without changing the basic semantics of how experts are mixed in the routing module. Based on this, it is easy to be compatible with the output of existing routing modules, and there is no need to retrain or relearn the new mixing weights of the routing module. The final processing result can be obtained only based on the normalization of the original weights of the routing module, which improves the compatibility and practicality of the solution.

[0109] In one possible embodiment, each expert layer in the MoE model may correspond to its own contribution strength threshold, or multiple expert layers may share the same contribution strength threshold. If each expert layer uses its own contribution strength threshold, a calibration text unit can be used to determine the contribution strength threshold for each expert layer. Specifically, Figure 7 This is a flowchart illustrating another inference method using a hybrid expert model provided in an exemplary embodiment of this disclosure. Figure 7 As shown, the steps to obtain the contribution intensity threshold for each expert layer are achieved by performing the following steps D1 to D4: Step D1: Obtain multiple candidate thresholds; Step D2: For each of the multiple candidate thresholds, prune each expert layer based on the calibration text and each candidate threshold to determine each calibration candidate expert before pruning and the calibration target expert after pruning. Step D3: Based on each calibration candidate expert before pruning and the calibration target expert after pruning, determine the mean square error and pruning rate corresponding to each candidate threshold; The mean square error represents the difference between the processing result of the calibration text before pruning for each expert layer and the processing result of the calibration text after pruning based on each candidate threshold; the pruning rate represents the ratio of experts in each expert layer to be pruned based on each candidate threshold. Step D4: Select multiple first candidate thresholds from the multiple candidate thresholds whose mean square error satisfies the error budget condition, and select the target candidate threshold corresponding to the maximum pruning rate from the multiple first candidate thresholds as the contribution intensity threshold for each expert layer.

[0110] Among them, pruning refers to using a contribution intensity threshold to prune multiple candidate experts in the expert layer.

[0111] The expert layer's processing result of the calibration text before pruning refers to the final processing result of the expert layer on the calibration text obtained by using the top-k calibration candidate experts selected based on the original weights to process the calibration text without pruning them, and by weighting the k processing results.

[0112] The processing result of the calibration text by the expert layer after pruning refers to using the candidate threshold to select the calibration target expert corresponding to the candidate threshold from the top-k calibration candidate experts, using the calibration target expert corresponding to the candidate threshold to process the calibration text, and obtaining the final processing result of the calibration text by the expert layer based on the processing result of the calibration target expert.

[0113] In this step, for each expert layer, multiple candidate thresholds can be obtained. For each candidate threshold, following the same steps as steps B1 to B2 above, target experts with expected contribution scores not lower than the candidate threshold are selected from multiple calibration candidate experts. The calibration text is then processed based on the calibration target experts corresponding to the candidate threshold. The processing result of the calibration text after pruning of the expert layer is obtained based on the processing result of the calibration target experts. Furthermore, the calibration text is processed based on multiple calibration candidate experts. The processing result of the calibration candidate experts is obtained based on the processing result of the multiple calibration candidate experts. The processing result of the calibration text before pruning of the expert layer is obtained. Further, the mean square error corresponding to the candidate threshold is calculated based on the difference between the two processing results before and after pruning of the expert layer. The pruning rate corresponding to the candidate threshold is calculated based on the change in the number of experts in the expert layer before and after pruning.

[0114] In one possible embodiment, the pruning of each expert layer based on the calibration text and each candidate threshold, determining each calibration candidate expert before pruning and the calibration target expert after pruning for each expert layer, includes: Based on the original weights of each expert in each expert layer for the calibration text, multiple calibration candidate experts before pruning are selected from multiple experts. Based on the static enhancement capabilities and original weights of these multiple calibration candidate experts, the expected contribution strength of these multiple calibration candidate experts to the calibration text is determined. The expected contribution intensity of the multiple calibration candidate experts to the calibration text is normalized to obtain the expected contribution score of the multiple calibration candidate experts. For each of the multiple candidate thresholds, based on each candidate threshold and the expected contribution score of the multiple calibration candidate experts, a mask for each calibration candidate expert is determined, so as to obtain the pruned calibration target expert in each expert layer based on the mask of the calibration candidate expert.

[0115] The implementation methods for screening multiple calibration candidate experts are the same as those for step 1021 above, the implementation methods for determining the expected contribution intensity of multiple calibration candidate experts are the same as those for step 1022 above, and the implementation methods for determining the expected contribution score and mask of calibration candidate experts are the same as those for steps B1 to B2 above. These will not be described in detail here.

[0116] In one possible embodiment, determining the mean square error and pruning rate corresponding to each candidate threshold based on each calibration candidate expert before pruning and the calibration target expert after pruning includes: The calibration text is processed based on multiple calibration candidate experts to obtain the first processing result of each expert layer on the calibration text, and the calibration text is processed based on the calibration target expert to obtain the second processing result of each expert layer on the calibration text. Based on the first and second processing results, the mean square error corresponding to each candidate threshold is determined; Based on the mask of the multiple calibration candidate experts, the pruning rate corresponding to each candidate threshold is determined.

[0117] In one possible example, for any candidate threshold, the processing result of the expert layer on the calibration text output before pruning can be calculated using the following formula (18):

[0118] in, This indicates the top-k calibration candidate experts in the expert layer before pruning, and their views on the calibration text. The processing results. This represents the processing result of the i-th calibration candidate expert among the top-k calibration candidate experts on the calibration text. This represents the original weight of the i-th calibration candidate expert.

[0119] And, using the following formula (19), the mean square error between the treatment results before and after pruning of the expert layer is calculated:

[0120] in, Indicates candidate threshold The corresponding mean squared error is calculated using the candidate threshold. For the first l After layer-by-layer pruning, the calibration target experts obtained from the calibration text x Processing results The results of the calibration text processing by the top-k calibration candidate experts before pruning. The relative mean square error between them. The calibration text may include multiple calibration text units. For each calibration text unit, the mean square error between the processing result of the calibration target expert on that calibration text unit and the mean square error between the top-k calibration candidate experts on that calibration text unit can be calculated. Based on this, multiple mean square errors corresponding to multiple calibration text units in the calibration text are obtained. This represents the expected value of multiple mean square errors corresponding to multiple calibration text units. The expected value of multiple mean square errors for multiple calibration texts can be calculated and used as a candidate threshold. Corresponding mean square error .

[0121] It should be noted that, in the embodiments of the present invention, the formulas for normalization and the mean square error formula introduce... The term can represent a numerically stable term, which can prevent instability caused by a denominator of zero or a value that is too small.

[0122] Furthermore, the pruning rate corresponding to the candidate threshold can be calculated using the following formula (20):

[0123] in, Indicates candidate threshold The corresponding pruning rate, Indicates based on candidate threshold When pruning the top-k calibration candidate experts, the expected value of the mask of the top-k calibration candidate experts. Specifically, for each candidate threshold, the mask of each calibration candidate expert among the top-k calibration candidate experts in the expert layer can be determined based on the candidate threshold by performing steps similar to steps B1 to B2 above.

[0124] Furthermore, regarding the first l Each candidate threshold in the expert layer , This is the set of candidate thresholds for the expert layer. After calculating the mean square error and pruning rate for each candidate threshold, the candidate threshold that satisfies the condition that the mean square error is not higher than the error budget and maximizes the pruning rate is selected from multiple candidate thresholds and is used as the final contribution intensity threshold of the expert layer.

[0125] For example, based on the error budget condition corresponding to the following formula (21), a set of thresholds including multiple first candidate thresholds can be selected from multiple candidate thresholds:

[0126] in, Indicates the first l Error budgeting for each expert layer. Based on the mean squared errors corresponding to multiple candidate thresholds, thresholds with mean squared errors not exceeding a certain threshold can be selected. Multiple first candidate thresholds.

[0127] In one possible example, the error budget for each expert level can be set differently for each expert level. Alternatively, a unified error budget could be set for multiple expert layers. .

[0128] Among them, the more relaxed the error budget, that is... The lower the value, the higher the pruning rate is usually obtained; the stricter the error budget, that is... The higher the value, the higher the output fidelity.

[0129] Furthermore, the target candidate threshold with the largest pruning rate can be selected as the contribution intensity threshold of the expert layer from multiple first candidate thresholds using the following formula (22):

[0130] in, This represents the contribution strength threshold of the expert layer, which is also the threshold for selecting target candidates. This represents the candidate threshold that maximizes the mean squared error. It should be noted that the calibration text does not need to be very large. In this embodiment of the invention, the default cache size for a single expert layer can be limited to a few, such as hundreds of tokens, thereby completing the search for the target candidate threshold at a relatively low cost.

[0131] Based on this embodiment, by further screening out the contribution intensity threshold of the maximum pruning rate from multiple first candidate thresholds whose mean square error meets the error budget condition, as much as possible more experts with small impact can be removed while ensuring that pruning of the expert layer does not affect the final output of the expert layer. This ensures the processing accuracy and reliability of the expert layer after pruning and improves the actual efficiency of pruning.

[0132] To more clearly describe the steps and flow of the reasoning method of this invention, as follows: Figure 8 As shown, the following is an implementation process in engineering, including the following steps 1 to 4: Step 1: Load the MoE model and iterate through all expert layers in the MoE model; like Figure 8 As shown, for a pre-trained MoE model, layer parameters can be read based on `gate_up_proj` (gated up projection component) and `down_proj` (down projection component). For example, for experts like Qwen3-MoE that use a SwiGLU structure, the engineering parameters of each expert layer can be loaded based on the one-to-one correspondence between the stored engineering parameters of each expert layer and the parameter matrices. The `gate_up_proj`, `down_proj`, and `post_attention_layernorm.weight` (post-attention layer normalized weight component) of each layer can be identified, and the matrix parameters of the expert layer can be obtained based on the one-to-one correspondence between the engineering parameters and the parameter matrices. In engineering implementation, the normalized scaling vector can be read based on the `post_attention_layernorm.weight` of the expert layer. .

[0133] Step 2: Calculate the static enhancement capabilities for each expert; like Figure 8As shown, in Phase One, offline static enhancement capabilities can be built for each expert. For example, expert... e The parameters are stored in the form of gate_up_proj[e] and down_proj[e]. gate_up_proj[e] can be divided into two halves along the first dimension. The gated branch weight matrix is ​​obtained by transposing the first half. The transpose of the latter half yields the activation branch weight matrix. The output projection matrix is ​​obtained by transposing down_proj[e]. If the first part of `gate_up_proj[e]` is `gate_proj_weight` and the second part is `up_proj_weight`, then correspondingly, as shown below... Figure 8 As shown, by iterating through multiple experts in each expert layer, and based on the following formulas (23) to (25), the matrix parameters of the experts are obtained by utilizing the mapping relationship between the engineering parameters and each parameter matrix: Formula (23) Formula (24) Formula (25) in, , , Experts e The activation branch weight matrix, the gated branch weight matrix, and the output projection matrix.

[0134] Furthermore, such as Figure 8 As shown, the static enhancement capability of the expert layer is obtained through formulas (1) to (3) and (11) in step A2 above, specifically including the experts in the expert layer. e Static enhancement capability Activate the static enhancement capability of the branch perspective Static enhancement capability of gated branch perspective This outputs amp_table[layer_id] (mixed precision configuration table [indexed by expert layer id]) and [expert_id] (expert id index).

[0135] Step 3: Collect calibration buffer; like Figure 8 As shown, a second stage of offline calibration is further performed. In order to search for the optimal threshold for each layer without repeating the full experiment, this invention performs forward computation on a small sample of calibration text and collects the following caches for each expert layer: topk_idx, gate, expert_outs, and full_out for each expert layer.

[0136] Where topk_idx represents the top-k expert ID for each token; gate represents the original weights of the top-k experts for each token; expert_outs represents the expert output vector for each slot (the expert activated by each token) for each token; and full_out represents the complete output of the token at the expert layer without pruning any experts.

[0137] Step 4: Search layer threshold; In the threshold set upper traversal The optimal threshold is selected based on the mean squared error and pruning rate. Based on this, a deployment configuration is generated, which may include, for example, static augmentation capabilities for deploying the trained MoE model, the optimal threshold, etc.

[0138] Step 5: Dynamic pruning during inference like Figure 8 As shown, the third stage of online inference is then performed. For each token, the routing module obtains the top-k candidate experts and their original weights; the expected contribution strength of each candidate expert is then calculated. Expected contribution score , mask It also forces the retention of the expert with the highest expected contribution score, and then performs renormalization and output mixing based on the original weights. The mixed output is only performed on the processing results of the retained target experts.

[0139] Furthermore, the following items can be statistically analyzed for each expert layer: total number of tokens; total number of selectable slots, which is the number of tokens multiplied by top-k, i.e., the number of candidate experts before pruning in each expert layer of the model; and the actual number of retained slots, i.e., the number of target experts after pruning in each expert layer. Based on this, the average pruning rate and average number of activated experts of the MoE model are calculated using the following formulas (26) and (27):

[0140]

[0141] in, This represents the average pruning rate across all expert layers of the model. This represents the average number of activated experts across all expert layers of the model. This indicates the number of target experts retained after pruning. This indicates the number of candidate experts before pruning. This represents the total number of tokens.

[0142] Based on this, the average pruning rate and average number of activated experts of the MoE model can be used to evaluate the actual acceleration benefits during online operation.

[0143] In this embodiment of the invention, during engineering deployment, the forward process of the expert layer of the model can be packaged: the original routing module is retained, and after the routing module obtains the top-k candidate experts and their original weights, the expected contribution scores can be further evaluated to filter out experts with actual effective contributions. Only the retained target experts participate in the final processing and combination, and the original forward logic is restored when exiting the context. Based on this, the present invention does not require modification of the MoE model's main parameters and does not require resaving the weights to complete runtime verification; if product deployment is required, the threshold table and similarity table can also be fixed as auxiliary configurations.

[0144] In summary, the present invention has at least the following beneficial effects: 1. Importance assessment can be completed before experts perform actual inference calculations: By combining the original weights of candidate experts with static enhancement capabilities, token-level importance capability assessments can be obtained without performing forward calculations by experts, thus making "early pruning" truly effective.

[0145] 2. Balancing token conditionality and expert structure differences: Unlike the pruning scheme that selects the top-k candidate experts based solely on the original weights, this invention considers both the "routing preference of the current token" and the "static output amplification capability of the expert," thus providing a more comprehensive and complete characterization of the expert's contribution capability.

[0146] 3. No retraining or backpropagation required: This invention relies only on the weight parameters of the pre-trained MoE model and a small amount of calibration text to complete offline static capability assessment and contribution strength threshold search. It has a low deployment threshold and is suitable for rapid integration after training.

[0147] 4. Compatible with mainstream SwiGLU MoE architecture: This invention clearly provides the expert parameter mapping method for the gate_up_proj+down_proj components of the Qwen3-MoE model, which can be directly applied to actual open source models.

[0148] 5. Maintain router semantic stability: Renormalize the original gate on a retained subset instead of relearning new hybrid rules, thus making it more compatible with existing inference frameworks.

[0149] Based on the above effects, the present invention has clear industrial applicability, specifically including the following practical scenarios: 1. Suitable for inference acceleration scenarios of large language models and multi-task generation models, especially suitable for MoE architecture with top-k routing.

[0150] 2. It does not rely on retraining and is suitable for rapid integration into existing inference frameworks, offline evaluation frameworks, and deployment pipelines.

[0151] 3. It can be directly used in commercial scenarios such as data center inference, private deployment, low-latency dialogue systems, code generation services, and knowledge question answering services.

[0152] 4. For industrial systems that require fine-grained balance between "accuracy, latency, and throughput", this invention provides two controllable solutions: a basic mode and a redundant sensing enhancement mode, which have high engineering implementation value.

[0153] The inference method of any hybrid expert model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the inference method of any hybrid expert model provided in this disclosure can be executed by a processor, such as by a processor executing the inference method of any hybrid expert model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0154] Figure 9 A block diagram of the inference apparatus for a hybrid expert model provided in an embodiment of this disclosure is shown. Figure 9 As shown, the device includes: The acquisition module 901 is used to acquire the data to be reasoned by the hybrid expert model. The data to be reasoned is text or multimedia data to be processed. The data to be reasoned includes multiple units, each unit being a word or a multimedia block. The hybrid expert model includes multiple expert layers, and each expert layer includes multiple experts. The reasoning module 902 is used to process the data to be reasoned through the multiple expert layers in sequence, and output the reasoning result of the data to be reasoned after the last expert layer has processed the data to be reasoned. Specifically, when processing the data to be reasoned through each expert layer, the reasoning module 902 is used for: For each of the multiple units, based on the original weights of each expert in each expert layer for each unit, multiple candidate experts are selected from the multiple experts in each expert layer. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts to each unit is determined; Based on the expected contribution intensity of the multiple candidate experts to each unit, the target expert corresponding to each unit is selected from the multiple candidate experts; The target expert corresponding to each unit processes each unit to obtain the processing result of each unit.

[0155] In one possible implementation, the apparatus further includes a determining module 903, which, when determining the static enhancement capabilities of multiple experts in each expert layer, is specifically used for: Determine the diagonal matrix of the normalized scaling vector for each expert layer; For each expert in each expert layer, the original static augmentation capability of each expert is determined based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert. The original static enhancement capabilities of multiple experts in each expert layer are normalized to obtain the static enhancement capabilities of those multiple experts.

[0156] In one possible implementation, the determining module 903, when determining the original static enhancement capability of each expert based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert, is specifically used for: Based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert, the static enhancement capability of the activation branch and the static enhancement capability of the gated branch of each expert are determined respectively. The static enhancement capabilities of the active branches and the gated branches of each expert are averaged to obtain the original static enhancement capability of each expert.

[0157] In one possible implementation, the inference module 902, when selecting the target expert corresponding to each unit from the multiple candidate experts based on the expected contribution intensity of each candidate expert to each unit, is specifically used for: The expected contribution intensity of the multiple candidate experts for each unit is normalized to obtain the expected contribution score of the multiple candidate experts for each unit. Based on the contribution intensity threshold of each expert layer and the expected contribution scores of the multiple candidate experts, a mask is determined for each candidate expert among the multiple candidate experts, so as to retain the target expert among the multiple candidate experts and prune the non-target experts among the multiple candidate experts except for the target expert based on the mask of each candidate expert; wherein, the mask of each candidate expert indicates whether to retain each candidate expert.

[0158] In one possible implementation, the inference module 902, when processing each unit based on the target expert corresponding to each unit to obtain the processing result of each unit, is specifically used for: Based on the mask of each candidate expert, the original weights of the multiple candidate experts are normalized to obtain the pruning weights of the multiple candidate experts. Each of the multiple candidate experts processes each unit individually, thus obtaining the processing results of the multiple candidate experts for each unit; Based on the pruning weights of the multiple candidate experts, the processing results of each candidate expert for each unit are weighted to obtain the processing results of each expert layer for each unit after pruning.

[0159] In one possible implementation, the device further includes a threshold acquisition module 904, which, when determining the contribution strength threshold for each expert layer, is specifically used for: Obtain multiple candidate thresholds; For each of the multiple candidate thresholds, each expert layer is pruned based on the calibration text and each candidate threshold to determine each calibration candidate expert before pruning and the calibration target expert after pruning. Based on each calibration candidate expert before pruning and the calibration target expert after pruning, the mean square error and pruning rate corresponding to each candidate threshold are determined; wherein, the mean square error represents the difference between the processing result of the calibration text before pruning of each expert layer and the processing result of the calibration text after pruning based on each candidate threshold; the pruning rate represents the ratio of experts in each expert layer to be pruned based on each candidate threshold; From these multiple candidate thresholds, select multiple first candidate thresholds whose mean squared error satisfies the error budget condition. From these multiple first candidate thresholds, select the target candidate threshold corresponding to the maximum pruning rate, which serves as the contribution intensity threshold for each expert layer.

[0160] In one possible implementation, when the threshold determination module 904 prunes each expert layer based on the calibration text and each candidate threshold, and determines each calibration candidate expert before pruning and the calibration target expert after pruning for each expert layer, it is specifically used for: Based on the original weights of each expert in each expert layer for the calibration text, multiple calibration candidate experts are selected from multiple experts. Based on the static enhancement capabilities and original weights of these multiple calibration candidate experts, the expected contribution strength of these multiple calibration candidate experts to the calibration text is determined. For each of the multiple candidate thresholds, based on each candidate threshold and the expected contribution score of the multiple calibration candidate experts, a mask for each calibration candidate expert is determined, so as to obtain the pruned calibration target expert in each expert layer based on the mask of the calibration candidate expert.

[0161] In one possible implementation, when the threshold determination module 904 determines the mean square error and pruning rate corresponding to each candidate threshold based on each calibration candidate expert before pruning and the calibration target expert after pruning, it is specifically used for: The calibration text is processed based on multiple calibration candidate experts to obtain the first processing result of each expert layer on the calibration text, and the calibration text is processed based on the calibration target expert to obtain the second processing result of each expert layer on the calibration text. Based on the first and second processing results, the mean square error corresponding to each candidate threshold is determined; Based on the mask of the multiple calibration candidate experts, the pruning rate corresponding to each candidate threshold is determined.

[0162] The hybrid expert model inference device provided in this embodiment of the invention acquires the data to be inferred from the hybrid expert model, processes the data sequentially through multiple expert layers in the hybrid expert model, and outputs the inference result of the data after processing by the last expert layer. Specifically, in each expert layer, for each unit, multiple candidate experts are selected from the multiple experts in that expert layer based on the original weights of each expert for that unit. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts for that unit is determined. Based on the expected contribution intensity of the multiple candidate experts for that unit, the target expert corresponding to that unit is selected from the multiple candidate experts. Based on this, the contributions and original weights of the candidate experts can be combined to accurately activate the effective contributing experts with high static enhancement capabilities among the candidate experts, and the unit is processed based on the target expert to obtain the processing result. This ensures that each unit is accurately processed by the experts who actually make effective contributions, thereby improving the accuracy of inference and avoiding the additional latency and memory bandwidth overhead caused by low-contribution experts.

[0163] It should be noted that the specific implementation of the inference device of the hybrid expert model in this disclosure corresponds to the specific implementation of the inference method of the hybrid expert model in this disclosure. The two can be referenced, combined and cited from each other. Moreover, the technical effects of the inference device of the hybrid expert model in this disclosure are similar to the technical effects of the inference method of the hybrid expert model in this disclosure. For details, please refer to the description of the inference method of the hybrid expert model. In order to reduce redundancy, it will not be described in detail.

[0164] Below, for reference Figure 10This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0165] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0166] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0167] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0168] The memory can store one or more computer programs and may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer programs may be stored on the computer-readable storage medium, and the processor can run the computer programs to implement the sound source localization methods of the various embodiments of this disclosure described above and / or other desired functions.

[0169] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0170] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0171] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0172] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0173] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer programs comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the reasoning methods of hybrid expert models according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0174] The computer program can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0175] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the reasoning methods of hybrid expert models according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0176] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0177] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0178] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0179] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0180] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0181] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0182] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0183] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A reasoning method of a hybrid expert model, characterized by, include: The hybrid expert model obtains the data to be reasoned, which is text or multimedia data to be processed. The data to be reasoned includes multiple units, each of which is a word or a multimedia block. The hybrid expert model includes multiple expert layers, and each expert layer includes multiple experts. The data to be reasoned is processed sequentially through the multiple expert layers, and the reasoning result of the data to be reasoned is output after the last expert layer processes the data to be reasoned. The processing of the data to be reasoned through each expert level includes: For each of the plurality of units, based on the original weights of each expert in each expert layer for each unit, multiple candidate experts are selected from the multiple experts in each expert layer. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts to each unit is determined. Based on the expected contribution intensity of the multiple candidate experts to each unit, the target expert corresponding to each unit is selected from the multiple candidate experts; Each unit is processed by the target expert corresponding to each unit to obtain the processing result of each unit. The reasoning result of the data to be reasoned includes the processing results of the multiple units.

2. The method of claim 1, wherein, It also includes the step of determining the static enhancement capabilities of multiple experts in each expert layer: Determine the diagonal matrix of the normalized scaling vector for each expert layer; For each expert in each expert layer, the original static enhancement capability of each expert is determined based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert. The original static enhancement capabilities of multiple experts in each expert layer are normalized to obtain the static enhancement capabilities of the multiple experts.

3. The method according to claim 2, characterized in that, The determination of the original static enhancement capability of each expert based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert includes: Based on the weight matrix of the activation branch, the weight matrix of the gated branch, the output projection matrix, and the diagonal matrix of each expert, the static enhancement capability of the activation branch of each expert and the static enhancement capability of the gated branch of each expert are determined respectively. The static enhancement capabilities of the activated branches and the gated branches of each expert are averaged to obtain the original static enhancement capability of each expert.

4. The method according to claim 1, characterized in that, The step of selecting the target expert for each unit from the multiple candidate experts based on their expected contribution strength to each unit includes: The expected contribution intensity of the multiple candidate experts to each unit is normalized to obtain the expected contribution score of the multiple candidate experts to each unit. Based on the contribution intensity threshold of each expert layer and the expected contribution scores of the plurality of candidate experts, a mask is determined for each candidate expert among the plurality of candidate experts, so as to retain the target expert among the plurality of candidate experts and prune the non-target experts among the plurality of candidate experts except the target expert; wherein, the mask of each candidate expert indicates whether to retain each candidate expert.

5. The method according to claim 4, characterized in that, The process of processing each unit based on the target expert corresponding to each unit to obtain the processing result of each unit includes: Based on the mask of each candidate expert, the original weights of the multiple candidate experts are normalized to obtain the pruning weights of the multiple candidate experts. Each of the multiple candidate experts processes each unit separately, thereby obtaining the processing results of the multiple candidate experts for each unit; Based on the pruning weights of the multiple candidate experts, the processing results of the multiple candidate experts on each unit are weighted to obtain the processing results of each expert layer on each unit after pruning.

6. The method according to claim 4, characterized in that, It also includes the step of obtaining the contribution strength threshold for each expert layer: Obtain multiple candidate thresholds; For each of the plurality of candidate thresholds, each expert layer is pruned based on the calibration text and each candidate threshold to determine each calibration candidate expert before pruning and the calibration target expert after pruning. Based on each calibration candidate expert before pruning and the calibration target expert after pruning, the mean squared error and pruning rate corresponding to each candidate threshold are determined; wherein, the mean squared error represents the difference between the processing result of the calibration text before pruning and the processing result of the calibration text after pruning based on each candidate threshold; the pruning rate represents the ratio of experts in each expert layer to be pruned based on each candidate threshold; From the plurality of candidate thresholds, a plurality of first candidate thresholds whose mean squared error satisfies the error budget condition are selected, and from the plurality of first candidate thresholds, a target candidate threshold corresponding to the maximum pruning rate is selected as the contribution intensity threshold of each expert layer.

7. The method according to claim 6, characterized in that, The step of pruning each expert layer based on the calibration text and each candidate threshold, and determining each calibration candidate expert before pruning and the calibration target expert after pruning, includes: Based on the original weights of each expert in each expert layer for the calibration text, multiple calibration candidate experts are selected from multiple experts. Based on the static enhancement capabilities and original weights of the multiple calibration candidate experts, the expected contribution strength of the multiple calibration candidate experts to the calibration text is determined. For each of the plurality of candidate thresholds, a mask for each calibration candidate expert is determined based on each candidate threshold and the expected contribution score of the plurality of calibration candidate experts, so as to obtain the pruned calibration target expert in each expert layer based on the mask of the calibration candidate expert.

8. The method according to claim 7, characterized in that, The step of determining the mean square error and pruning rate corresponding to each candidate threshold based on each calibration candidate expert before pruning and the calibration target expert after pruning includes: The calibration text is processed based on multiple calibration candidate experts to obtain a first processing result for each expert layer on the calibration text, and the calibration text is processed based on the calibration target expert to obtain a second processing result for each expert layer on the calibration text. Based on the first processing result and the second processing result, the mean square error corresponding to each candidate threshold is determined. Based on the masks of the multiple calibration candidate experts, the pruning rate corresponding to each candidate threshold is determined.

9. A reasoning device using a hybrid expert model, characterized in that, include: The acquisition module is used to acquire the data to be reasoned by the hybrid expert model. The data to be reasoned is text or multimedia data to be processed. The data to be reasoned includes multiple units, each unit being a word or a multimedia block. The hybrid expert model includes multiple expert layers, and each expert layer includes multiple experts. The reasoning module is used to process the data to be reasoned through the multiple expert layers in sequence, and output the reasoning result of the data to be reasoned after the last expert layer has processed the data to be reasoned. Specifically, when the reasoning module processes the data to be reasoned through each expert layer, it is used for: For each of the plurality of units, based on the original weights of each expert in each expert layer for each unit, multiple candidate experts are selected from the multiple experts in each expert layer. Based on the static enhancement capabilities and original weights of the multiple candidate experts, the expected contribution intensity of the multiple candidate experts to each unit is determined. Based on the expected contribution intensity of the multiple candidate experts to each unit, the target expert corresponding to each unit is selected from the multiple candidate experts; Each unit is processed by the target expert corresponding to each unit to obtain the processing result of each unit. The reasoning result of the data to be reasoned includes the processing results of the multiple units.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-8.

11. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

12. A computer program, comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.

Images