Hybrid expert model reasoning method and apparatus

Abstract

Embodiments of the present disclosure disclose a hybrid expert model inference method and device. The specific implementation of the method comprises: obtaining a hybrid expert model and an expert selection prediction model for the hybrid expert model, wherein the hybrid expert model comprises multiple layers, and each layer comprises multiple expert models; determining, based on a token and a context of the token, a target expert model to be activated in different layers of the hybrid expert model by the expert selection prediction model; constructing a sub-model based on the target expert model; and inferring a context of the token by the sub-model based on the token and the context of the token. The implementation improves inference efficiency and accuracy while keeping the controllable computing cost.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of machine learning technology, and more specifically to hybrid expert model inference methods and apparatus. Background Technology

[0002] Large language models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing tasks. However, as these models grow in size and complexity, the computational resources required for inference become increasingly expensive. Mixture-of-Experts (MoE) architectures have emerged as a promising approach, increasing model capacity while maintaining manageable computational costs by conditionally activating only a subset of parameters during inference.

[0003] The MoE model divides its parameters into multiple "expert" networks and employs a gating mechanism to determine which experts should process a given input. While this approach reduces the amount of active computation per token, the MoE model still faces significant inference challenges, including the high memory requirements for storing all experts, the computational overhead of the gating mechanism, and the bottleneck of token generation order.

[0004] Traditional inference optimization techniques, such as quantization, knowledge distillation, and pruning, often sacrifice model quality or require extensive retraining. Therefore, inference optimization techniques specifically tailored to the structural characteristics of MoE models are needed to improve inference efficiency without sacrificing model quality. Summary of the Invention

[0005] Embodiments of this disclosure present a hybrid expert model reasoning method and apparatus.

[0006] In a first aspect, embodiments of this disclosure provide a hybrid expert model reasoning method, comprising: obtaining a hybrid expert model and an expert selection prediction model for the hybrid expert model, wherein the hybrid expert model includes multiple layers, each layer including multiple expert models; determining the target expert model to be activated in different layers of the hybrid expert model based on the token and the context of the token through the expert selection prediction model; constructing a sub-model based on the target expert model; and reasoning the context of the token through the sub-model based on the token and the context of the token.

[0007] In some embodiments, the method further includes: inferring multiple candidate tokens based on the context of the token using a lightweight token predictor; for each candidate token, inferring the context of the candidate token in parallel using a partially activated hybrid expert model based on the context of the token and the candidate token; verifying the correct candidate token from the multiple candidate tokens using a fully activated hybrid expert model; and outputting the context of the correct candidate token.

[0008] In some embodiments, the process of inferring the context of a candidate token in parallel using a partially activated hybrid expert model, based on the context of the token and the candidate token, includes: determining candidate expert models to be activated in different layers of the hybrid expert model using an expert selection prediction model based on the context of the token and the candidate token; constructing candidate sub-models based on the candidate expert models; assigning the candidate token to the candidate sub-models; and having the candidate sub-models infer the context of the candidate token in parallel based on the context of the token and the candidate token.

[0009] In some embodiments, verifying the correct candidate token from multiple candidate tokens using a fully activated hybrid expert model includes: inferring the actual next token based on the context of the token using a fully activated hybrid expert model; matching the next token with each candidate token, and selecting the successfully matched candidate token as the correct candidate token.

[0010] In some embodiments, the method further includes: in response to a matching failure, inferring and outputting the context of the next token based on the next token using a fully activated hybrid expert model.

[0011] In some embodiments, the method further includes: obtaining information about the expert models activated during the process of reasoning about the following text of the target token sequence through a hybrid expert model based on the target token sequence; generating training samples based on the information and the target token sequence; and training an expert selection prediction model based on the training samples.

[0012] In some embodiments, constructing a sub-model based on the target expert model includes: loading relevant parameters of the target expert model into memory; and constructing a sub-model based on the relevant parameters in memory.

[0013] In some embodiments, based on the token and its context, the target expert model to be activated in different layers of the hybrid expert model is determined by an expert selection prediction model, including: based on the token and its context, the activation probability of each expert model in each layer is output by the expert selection prediction model; and the expert model in each layer whose activation probability exceeds a predetermined threshold is determined as the target expert model to be activated.

[0014] In some embodiments, the method further includes dynamically adjusting a predetermined threshold based on available computing resources and / or inference accuracy requirements.

[0015] In some embodiments, based on the token and its context, the target expert models to be activated in different layers of the hybrid expert model are determined by an expert selection prediction model, including: based on the token and its context, the activation probability of each expert model in each layer is output by an expert selection prediction model; and a predetermined number of expert models with the highest activation probability in each layer are selected as the target expert models to be activated.

[0016] In some embodiments, the method further includes adjusting the number of candidate tokens based on at least one of the following: network throughput, computing resource utilization, and the amount of available computing resources.

[0017] Secondly, embodiments of this disclosure provide a hybrid expert model inference apparatus, comprising: an acquisition unit configured to acquire a hybrid expert model and an expert selection prediction model for the hybrid expert model, wherein the hybrid expert model includes multiple layers, each layer including multiple expert models; a determination unit configured to determine, based on a token and its context, a target expert model to be activated in different layers of the hybrid expert model using the expert selection prediction model; a construction unit configured to construct a sub-model based on the target expert model; and an inference unit configured to infer the context of the token from the sub-model based on the token and its context.

[0018] In some embodiments, the apparatus further includes a spatial reasoning unit configured to: infer multiple candidate tokens based on the context of the token using a lightweight token predictor; for each candidate token, infer the context of the candidate token in parallel using a partially activated hybrid expert model based on the context of the token and the candidate token; verify the correct candidate token from the multiple candidate tokens using a fully activated hybrid expert model; and output the correct context of the candidate token.

[0019] In some embodiments, the spatial reasoning unit is further configured to: determine candidate expert models to be activated in different layers of the hybrid expert model based on the context of the token and the candidate token using an expert selection prediction model; construct candidate sub-models based on the candidate expert models; assign the candidate token to the candidate sub-models, and have the candidate sub-models infer the context of the candidate token in parallel based on the context of the token and the candidate token.

[0020] In some embodiments, the spatial reasoning unit is further configured to: infer the actual next token based on the context of the token using a fully activated hybrid expert model; match the next token with each candidate token, and select the successfully matched candidate token as the correct candidate token.

[0021] In some embodiments, the spatial reasoning unit is further configured to: in response to a matching failure, infer and output the context of the next token based on the next token using a fully activated hybrid expert model.

[0022] In some embodiments, the apparatus further includes a training unit configured to: acquire information about expert models activated during the process of reasoning about the following text of a target token sequence using a hybrid expert model based on the target token sequence; generate training samples based on the information and the target token sequence; and train an expert selection prediction model based on the training samples.

[0023] In some embodiments, the building unit is further configured to: load relevant parameters of the target expert model into memory; and build a sub-model based on the relevant parameters in memory.

[0024] In some embodiments, the determining unit is further configured to: based on the token and the context of the token, output the activation probability of each expert model in each layer through expert selection prediction model; and determine the expert model in each layer whose activation probability exceeds a predetermined threshold as the target expert model to be activated.

[0025] In some embodiments, the apparatus further includes a parameter tuning unit configured to dynamically adjust a predetermined threshold based on available computing resources and / or inference accuracy requirements.

[0026] In some embodiments, the determining unit is further configured to: based on the token and the context of the token, output the activation probability of each expert model in each layer by selecting an expert prediction model; and select a predetermined number of expert models with the highest activation probability in each layer as target expert models to be activated.

[0027] In some embodiments, the apparatus further includes a parameter tuning unit configured to adjust the number of candidate tokens based on at least one of the following: network throughput, computing resource utilization, and the amount of available computing resources.

[0028] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first or second aspects.

[0029] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0030] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0031] The hybrid expert model inference method and apparatus provided in the embodiments of this disclosure have significant advantages, including reducing memory requirements by loading only the necessary experts, reducing computational costs through optimized expert selection, and improving inference throughput through parallel processing of speculative tokens. By fusing temporal speculative sampling and spatial speculative sampling, the efficiency and quality of model inference are improved.

[0032] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0033] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied; Figure 2 This is a flowchart of an embodiment of the hybrid expert model reasoning method according to the present disclosure; Figure 3 This is a schematic diagram of an application scenario of the hybrid expert model reasoning method according to this disclosure; Figure 4 This is a flowchart of yet another embodiment of the hybrid expert model reasoning method according to the present disclosure; Figure 5 This is a schematic diagram of the structure of one embodiment of the hybrid expert model inference apparatus according to the present disclosure; Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0034] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] Figure 1An exemplary system architecture 100 is shown, in which embodiments of the hybrid expert model inference method or hybrid expert model inference apparatus of this disclosure can be applied.

[0037] like Figure 1 As shown, the system architecture 100 may include a server and multiple computing nodes. The server is responsible for storing and updating the parameters of models such as hybrid expert models and expert selection prediction models, while the computing nodes are responsible for model training and model inference.

[0038] The specific process of model training is as follows: 1. Distribute the training data (sample set) evenly to different computing nodes.

[0039] 2. Divide the model parameters into segments and store them on different servers.

[0040] 3. Computation Node: Reads a batch of training data, pulls the latest parameters from the server, calculates the gradient, and uploads it to different servers according to the data slices.

[0041] 4. Server: Receives gradients uploaded by computing nodes and updates parameters according to the optimization algorithm. Depending on whether the server needs to wait for gradients from all computing nodes for each parameter update, there are two mechanisms: synchronous training and asynchronous training.

[0042] The specific process of model inference is as follows: 1. For a given token, predict multiple candidate tokens using a lightweight model.

[0043] 2. Based on the tokens and each candidate token, predict which experts will be activated in each layer of the MoE model.

[0044] 3. For each candidate token, construct a sub-model containing only the activated experts, and assign the sub-model and the corresponding candidate token to different computing nodes.

[0045] 4. Different computing nodes, based on the assigned candidate tokens, infer the pre-computation result of the next token in parallel through sub-models.

[0046] 5. When the actual next token is determined, use the corresponding pre-calculated result.

[0047] The MoE model can be trained using System Architecture 100. In the MoE model, each layer consists of a set of expert networks (specialized neural network modules) and a gating network that determines which experts should handle a given input. During forward propagation, the gating network assigns routing probabilities to each expert, and typically, for each token, only the top k experts with the highest probabilities are activated. This sparse activation pattern is key to the efficiency of the MoE model because it allows the model to have a large number of parameter counts while using only a small fraction of the parameters for any given input.

[0048] It should be noted that the load balancing method for distributed model training provided in the embodiments of this disclosure can be executed by a server. Accordingly, the load balancing device for distributed model training can be located in the server. No specific limitations are made here.

[0049] It should be understood that Figure 1 The number of servers and compute nodes shown is merely illustrative. Depending on implementation needs, any number of servers and compute nodes can be used.

[0050] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of the hybrid expert model inference method according to the present disclosure. The hybrid expert model inference method includes the following steps: Step 201: Obtain the hybrid expert model and the expert selection prediction model for the hybrid expert model.

[0051] In this embodiment, the execution entity of the hybrid expert model inference method (e.g.) Figure 1 The server shown can obtain the hybrid expert model and the expert selection prediction model based on the hybrid expert model from a third-party server via wired or wireless connection. Alternatively, the hybrid expert model and the expert selection prediction model based on the hybrid expert model can be trained via computing nodes. The hybrid expert model comprises multiple layers, with each layer containing multiple expert models.

[0052] The expert selection prediction model can predict which expert models in each layer of the hybrid expert model should be activated given an input token sequence. The core of the expert selection prediction model is that the expert activation patterns in the MoE model are not random, but rather influenced by the input context and the current token being processed.

[0053] The expert selection prediction model can be a lightweight prediction model that, given the context and the current token, estimates the probability of activation for each expert model at each layer of the MoE model. This expert selection prediction model can be trained on trajectories collected from the full MoE model during inference on representative data.

[0054] For example, the MoE model currently has three layers, with three expert models in each layer, numbered as follows: First layer: 1 2 3 Second layer: 4 5 6 Third layer: 7 8 9 The expert selection prediction model can select at least one expert model to activate from each layer for the current token, for example, selecting to activate 1, 5, and 9.

[0055] The expert selection prediction model outputs the activation probability distribution of each layer of expert models. Based on these activation probabilities, a sub-model is constructed that contains only expert models whose activation probabilities exceed a configurable threshold. This sub-model can be prefetched and loaded into memory before actual inference begins, significantly reducing memory usage compared to loading the entire MoE model.

[0056] Optionally, the expert selection prediction model can be a Transformer network that takes the current token and its preceding embedding as input and outputs the activation probabilities of the expert model at each layer. This model can be trained using a cross-entropy loss between the predicted activation probabilities of the expert model and the actual activation probabilities observed when running the full MoE model.

[0057] Step 202: Based on the token and the context of the token, determine the target expert model to be activated in different layers of the hybrid expert model through the expert selection prediction model.

[0058] In this embodiment, a token refers to the smallest unit of the input sequence (e.g., a word or a subword). Token activation is a core technology of MoE, referring to the mechanism by which the model dynamically selects some expert models to process the current input token during inference. Essentially, it achieves low-cost deployment of trillion-parameter models through sparse computation.

[0059] For the token and its context, the expert selection prediction model outputs the activation probability of each layer of expert models. Expert models with activation probabilities greater than a predetermined threshold can be selected as target expert models. Alternatively, a predetermined number of expert models with the highest activation probabilities can be selected as target expert models within each layer.

[0060] In existing technologies, the input to expert-selected prediction models is usually only a token, while this application can also input the context of the token. For example, when it is necessary to predict the next token of "I plan to eat rice", "rice" is the current token, and "I plan to eat" is the context of the current token.

[0061] Step 203: Construct a sub-model based on the target expert model; In this embodiment, a sub-model containing only the target expert model is constructed based on these activation probabilities. This sub-model can be prefetched and loaded into memory before actual inference begins, significantly reducing memory usage compared to loading the entire MoE model.

[0062] Step 204: Based on the token and its preceding context, infer the following context of the token through the sub-model.

[0063] In this embodiment, the sub-model has the same functionality as the LLM, capable of inferring the following text of a token based on the token and its preceding text. For example, when predicting the next token after "I plan to eat rice," "rice" is the current token, and "I plan to eat" is the preceding text. The following text for the current token "rice" can be words that can form food names with "rice," such as "rice," "noodles," "rice noodles," or "flower candy."

[0064] The process of expert selection prediction and sub-model construction described above can be summarized by the following algorithm 1: Requirements: C above, current token t, complete MoE model M, expert selection prediction model P, activation threshold τ; Ensure: Optimized sub-model S; 1: Obtain the expert activation probability: A = P(C, t); 2: For each layer in the MoE model l Repeat the following operations: 3: Select an expert model with activation probability > τ: E l = {e | A [ l , e] > τ}; 4: End the loop; 5: The construction only contains each layer l Expert Model E l Sub-model S; 6: Return to S.

[0065] The method provided in the above embodiments of this disclosure optimizes the inference of the MoE model by integrating expert-selected prediction models, thereby improving inference efficiency and saving memory space.

[0066] In some optional implementations of this embodiment, the method further includes: inferring multiple candidate tokens based on the context of the token using a lightweight token predictor; for each candidate token, inferring the context of the candidate token based on the context of the token and the candidate token using a partially activated hybrid expert model; verifying the correct candidate token from the multiple candidate tokens using a fully activated hybrid expert model; and outputting the context of the correct candidate token.

[0067] This application introduces a spatial inference sampling method based on the concept of speculative decoding, where future tokens are predicted using a lightweight model to accelerate inference. However, traditional speculative decoding methods are primarily time-dependent, focusing on predicting a set of tokens in advance. The spatial inference sampling method in this application distributes speculative computation across different sub-models for parallel processing. This approach is particularly suitable for MoE models because it simultaneously leverages sparse expert activation patterns and the probabilistic nature of next token prediction.

[0068] In one embodiment, spatial inference sampling works as follows: First, a lightweight model (which could be a small language model or a dedicated next token predictor) generates multiple candidate tokens (i.e., the next token of the current token, also known as "the next to the current token"). Each candidate token has an associated generation probability, and a predetermined number of candidate tokens can be selected in descending order of generation probability.

[0069] For each candidate token, the expert selection prediction model determines which expert models might need to be activated if that candidate token is generated.

[0070] Based on these predicted expert models to be activated, a sub-model containing only the expert models required for each candidate token is constructed and loaded into memory.

[0071] Candidate tokens are assigned to these sub-models, and inference is performed in parallel to obtain pre-computed results.

[0072] When the actual next token is determined (either by the MoE model itself or by verification with the full model), the system retrieves the corresponding pre-calculated results, avoiding redundant calculations.

[0073] While spatial inference sampling focuses on distributing computation across different sub-models for parallel processing, it can be effectively combined with temporal inference sampling to further accelerate inference.

[0074] In one embodiment, the system first generates a sequence of candidate tokens (temporal inference) using a lightweight model. Then, for each candidate token in the sequence, spatial inference sampling is applied to distribute computation across an optimized sub-model.

[0075] This ensemble approach can significantly reduce end-to-end inference latency by predicting multiple tokens in advance and optimizing the computation of each predicted token.

[0076] The process of spatial inference sampling is demonstrated using Algorithm 2: Requirements: C (as stated above), complete MoE model M, lightweight prediction model L, expert-selected prediction model P, number of candidates k Ensure: the next token tnext and the updated above. 1: Generate candidate tokens using a lightweight model: 2: Tcandidates = {(t1, p1), (t2, p2), . . . , (tk, pk)} = L (C) 3: For each candidate token ti in Tcandidates, perform the following operations in a loop: 4: Predict the expert model to be activated: Ai = P (C, ti) 5: Constructing a sub-model Si based on Ai 6: The hidden state of the pre-computed candidate token (i.e., the next token, hereinafter referred to as below): Hi = Si (C, ti) 7: End the loop 8: Generate the actual next token using the complete MoE model or verification: tnext = M (C) 9: If tnext matches any ti in Tcandidates, then do the following: 10: Retrieve the hidden state Hi of the pre-computed candidate tokens. 11: Use tnext and Hi to update the above text 12: Otherwise 13: The following section describes the computation of candidate tokens using the full MoE model. 14: Update the above text 15: Termination Condition 16: Return to tnext and update the previous text. In some optional implementations of this embodiment, the inference of the context of the candidate token based on the context of the token and the candidate token by using a partially activated hybrid expert model includes: determining the candidate expert models to be activated in different layers of the hybrid expert model by using an expert selection prediction model based on the context of the token and the candidate token; constructing candidate sub-models based on the candidate expert models; and inferring the context of the candidate token by using the candidate sub-models based on the context of the token and the candidate token.

[0077] The process of constructing sub-models by selecting a prediction model through experts has already been introduced in the previous text, and will not be repeated here.

[0078] In some optional implementations of this embodiment, the correct candidate token is verified from multiple candidate tokens by using a fully activated hybrid expert model, including: inferring the actual next token based on the context of the token using a fully activated hybrid expert model; matching the next token with each candidate token, and taking the successfully matched candidate token as the correct candidate token.

[0079] Speculative decoding requires two models: a lightweight token predictor to quickly generate candidate tokens, and a full "hybrid expert model" to check the correctness of these candidate tokens. If the verification passes, the candidate tokens are accepted, saving time spent on multiple back-and-forth confirmations.

[0080] In some optional implementations of this embodiment, the method further includes: in response to a matching failure, inferring and outputting the context of the next token based on the next token using a fully activated hybrid expert model.

[0081] If verification fails, start generating again from the point of failure.

[0082] In some optional implementations of this embodiment, the method further includes: obtaining information about the expert models activated during the process of reasoning about the following text of the target token sequence through a hybrid expert model based on the target token sequence; generating training samples based on the information and the target token sequence; and training an expert selection prediction model based on the training samples.

[0083] In the process of reasoning about the target token sequence using a hybrid expert model, each activated expert model layer can be labeled as a positive sample, and each inactive expert model layer can be labeled as a negative sample. The expert selection prediction model is then trained in a supervised manner.

[0084] In some optional implementations of this embodiment, constructing a sub-model based on the target expert model includes: loading the relevant parameters of the target expert model into memory; and constructing the sub-model based on the relevant parameters in memory. This eliminates the need to load the entire MoE into memory; only the expert model that needs to be activated needs to be loaded, thus saving memory space.

[0085] In some optional implementations of this embodiment, the target expert models to be activated in different layers of the hybrid expert model are determined by an expert selection prediction model based on the token and its context. This includes: outputting the activation probability of each expert model in each layer by an expert selection prediction model based on the token and its context; and determining the expert models in each layer whose activation probability exceeds a predetermined threshold as the target expert models to be activated.

[0086] The activation threshold for the prediction model selected by experts can be dynamically adjusted based on resource constraints and accuracy requirements.

[0087] The expert selection prediction model actually provides an array of activation probabilities for each expert model in each layer. For example, the MoE model now has three layers, with three expert models in each layer, numbered as follows: First layer: 1 2 3 Second layer: 4 5 6 Third layer: 7 8 9 In this scenario, the activation probabilities given by the prediction model might be: 0.5, 0.3, 0.2 0.7, 0.1, 0.2 0.4, 0.5, 0.1 If the activation threshold is set to 0.5, then the selected experts will be... 0.5, -, - 0.7, -, - -, 0.5, - It is equivalent to activating 1, 4, 8 One more point to add: if the activation probability of each expert model in the current layer is lower than the activation threshold, then the expert model with the highest activation probability in that layer is selected.

[0088] The purpose of dynamically adjusting the activation threshold is to control which expert models are included in the inference sub-model at each layer. This directly provides a means of parameter tuning between high accuracy and low memory usage, which is a common requirement in real-world deployments. Low threshold → More expert models activated → Higher accuracy → Higher resource consumption; High threshold → fewer expert models activated → less memory saved → accuracy may decrease.

[0089] For example, autonomous driving vehicle decision-making systems need to complete environmental perception reasoning (such as identifying pedestrians, traffic lights, and obstacles) within millisecond-level response time. Their operating environment is dynamically affected by the computing power of the onboard chip (such as a GPU with a peak computing power of 200 TOPS) and the complexity of real-time road conditions. This system employs a 6-layer expert decision-making model, with each layer containing 12 expert networks, and the activation threshold is adjusted in conjunction with the vehicle's state.

[0090] When the vehicle is traveling at a constant speed on a highway (simple road conditions, decision delay tolerance ≤100ms), and the onboard chip utilization rate is less than 40%, the activation threshold is set to 0.4. Activating 8-10 experts per layer improves environmental perception accuracy, achieving a 98% accuracy rate in identifying distant obstacles (such as a disabled vehicle 500 meters away), thus reserving sufficient reaction time for the decision-making system.

[0091] When a vehicle enters a congested urban area (where road conditions are complex and frequent decisions are required) or the chip utilization rate exceeds 80%, the activation threshold is increased to 0.6. Activating 4-5 experts per layer reduces memory usage by 35% and compresses inference latency to within 50ms. Although the accuracy of recognizing blurry targets (such as pedestrians in low light conditions on rainy days) decreases by 2%-3%, the reliability of core decision-making functions is ensured by prioritizing the activation of experts related to "obstacle avoidance" and "traffic light recognition".

[0092] In some optional implementations of this embodiment, the method further includes dynamically adjusting a predetermined threshold based on available computing resources and / or inference accuracy requirements. If computing resources are abundant, or the inference accuracy requirement is high, a lower predetermined threshold can be set to allow more expert models to participate in inference, which can improve inference accuracy but consumes more computing resources. Therefore, when computing resources are scarce, a higher predetermined threshold can be set to minimize the number of expert models participating in inference.

[0093] In some optional implementations of this embodiment, based on the token and its context, the target expert models to be activated in different layers of the hybrid expert model are determined by an expert selection prediction model, including: based on the token and its context, the activation probability of each expert model in each layer is output by an expert selection prediction model; and a predetermined number of expert models with the highest activation probability in each layer are selected as the target expert models to be activated.

[0094] The effectiveness of expert-selected predictions depends on the accuracy of the predictive model. In practice, to address prediction errors, it may be beneficial to include a slightly larger ensemble of experts than a rigorously selected predictive model, trading some memory efficiency for higher accuracy.

[0095] In existing technologies, the expert model with the top 1 activation probability in each layer is typically selected. However, the accuracy of the top 1 expert model is not necessarily 100%, while the accuracy of the top 5 expert models can be very high. This means that the original algorithm design selects one expert model for each layer, but in reality, this expert selection prediction model can be chosen from the top N models. For example, the top 2 model can be selected for each layer.

[0096] Based on the example above, the activation probabilities given by the expert's prediction model might be: 0.5, 0.3, 0.2 0.7, 0.1, 0.2 0.4, 0.5, 0.1 The standard approach in existing technologies is to select only the top 1 candidate for each layer: 1 for the first layer, 4 for the second, and 8 for the third, constructing activation paths of 1, 4, and 8. However, due to the limited accuracy of the prediction model, it cannot reach 100%. Therefore, the prediction results of top N are used, such as the prediction results of top 2. In this case, the first layer might select 1 and 2, the second layer might select 4 and 6, and the third layer might select 7 and 8. This way, the accuracy of top 2 can be very close to 100%. Relatively speaking, using top N candidates increases memory usage, but it is still more economical than loading all expert models.

[0097] In some optional implementations of this embodiment, the method further includes: adjusting the number of candidate tokens based on at least one of the following information: network throughput, computing resource utilization, and the number of available computing resources.

[0098] The number K of candidate tokens in spatial inference sampling can be adjusted based on available computing resources, the number of nodes, network bandwidth, etc.

[0099] When K is small, reasoning is fast but not accurate enough; when K is large, it is more redundant and increases throughput, but consumes more resources.

[0100] For example, this solution can be deployed in a distributed cluster containing 1-32 compute nodes, interconnected via 100Gbps Ethernet. The system includes a node status monitoring module (collecting the number of online nodes N and the average bandwidth between nodes B), a token scheduling module, and a distributed inference engine. The token scheduling module calculates the number of candidate tokens K based on N and B, using the formula: K = min( N×B / 10 , 64), where To round down, 10 represents the bandwidth factor (unit: Gbps).

[0101] When the number of online nodes N=32 and the average bandwidth B=100Gbps, K=32. At this time, each node is assigned one candidate token, and the throughput is improved through parallel inference. It can process 8000 text sequences per second, which is 95% higher than when N=16, but the network transmission volume increases to 8GB / s.

[0102] When a node failure results in N=8 and B=50Gbps, K=4. The system automatically reduces the number of candidate tokens to lower the network load. At this point, the throughput drops to 3500 tokens / second, but the data transmission latency between nodes decreases from 20ms to 8ms.

[0103] When network congestion results in B=10Gbps (N=16), K=1. By avoiding bandwidth bottlenecks through serial inference, although the throughput drops to 800 packets / second, the inference success rate remains 100% (no data packet loss).

[0104] In a distributed scenario, the linkage adjustment of the K value with the number of nodes and network bandwidth not only avoids node idleness (load balancing issues when K < N) but also prevents inference failures caused by bandwidth overload, thus improving the availability of the framework by 60% in scenarios where nodes are dynamically online or offline.

[0105] Based on the sub-models required for the currently selected inference (e.g., 1, 4, 8), the system iterates through the available computing nodes to find the computing node with the highest network throughput (i.e., the one with the lowest resource utilization and sufficient GPU memory) and assigns the task to it. If the network throughput, computing resource utilization, and the number of available computing resources can support parallel inference across multiple computing nodes, the number of candidate tokens can be increased. For example, if there are currently 6 available computing nodes, the number of candidate tokens can be set to 6.

[0106] For example, in an intelligent translation system supporting translation between 20 languages, its MoE model contains 10 expert layers, with 8 expert networks in each layer (optimized for the grammatical rules and semantic features of different language pairs). The system employs a Top-K expert selection strategy, dynamically adjusting the K value to balance translation accuracy and memory usage.

[0107] When the system processes low-resource language pairs (such as Swahili-Chinese) translation, due to limited training data, the prediction model's matching accuracy with experts is only 75%. In this case, a Top-2 selection mode is activated: the top two experts in each layer are selected based on their matching scores to participate in inference. For example, in the "verb tense conversion layer," the prediction model prioritizes recommending "African language tense experts" (matching score 0.82) while also incorporating "low-resource language adaptation experts" (matching score 0.79), with both working together to correct the translation results. Actual testing shows that compared to the Top-1 mode, the BLEU score for low-resource language pairs improves by 4.2%, the probability of missing key semantics (such as tense differences and honorific expressions) decreases by 60%, and the memory usage per layer only increases by 12% (from 150MB with a single expert to 168MB with two experts).

[0108] When dealing with high-resource language pairs (such as English-Chinese), the prediction model achieves an accuracy of 92%. At this point, it automatically switches to Top-1 mode, activating only the expert with the highest matching score. While ensuring a stable translation BLEU value (≥45), the memory usage of each layer is controlled within 150MB to meet the lightweight deployment requirements of mobile terminals.

[0109] Systems that apply hybrid expert model reasoning methods include: The expert selection prediction module analyzes the context and the current token to predict the probability of expert model activation.

[0110] The submodel construction module is used to build optimized submodels based on the prediction results, containing only the necessary expert models.

[0111] The spatial speculative sampling module is used to generate candidate next tokens and distribute them to all sub-models for parallel processing.

[0112] The verification module, also known as the Speculative Acceptance Evaluator, is used to determine the actual next token and select the appropriate preprocessed result.

[0113] The memory management module is used to efficiently load and manage expert models based on prediction-based activation methods.

[0114] See also Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of the hybrid expert model reasoning method according to this embodiment. Figure 3 In application scenarios, The current token T 1t Input a candidate token generator, output a sequence of candidate tokens {T} 1t+1 T 1t+2 , ...T 1t+n The current token and candidate token sequences are fed into the Spatial Speculative Sampling Predictor, which outputs the distribution {D} of the expert models to be activated. 1t， D 1t+1 D1t+2 , ...D 1t+n}, where D 1t This indicates that the token T is being used as a reference. 1t Which expert models need to be activated, D 1t+1 D 1t+2 , ...D 1t+n This indicates that for the candidate token sequence {T} 1t+1 T 1t+2 , ...T 1t+n The SpatialSpeculative Sampling Predictor is a scheduler / process module used to execute the spatial sampling process. It calls the expert selection predictor to determine the subset of experts to be loaded for each candidate token. The Submodel Scheduler is responsible for "scheduling submodels," that is, allocating the constructed submodels to computing nodes and managing parallel inference. The memory manager, in addition to including the Offline Submodel Distribution Preload library, also performs run-time expert loading / unloading based on expert predictions. Only the expert models that need to be activated are loaded from the Offline Submodel Distribution Preload library to form submodels for different tokens. These submodels are then allocated to different computing nodes, and multiple computing nodes infer the context of different tokens in parallel to obtain the output results. The "submodel scheduler" and "memory manager" are logically distinct: one is responsible for scheduling computing resources, and the other is responsible for memory resource scheduling. The speculative acceptance evaluator verifies which candidate token is the actual token. The text following the actual token is then used as the final output.

[0115] Further reference Figure 4 This illustrates a flow 400 of another embodiment of the hybrid expert model inference method. Flow 400 of this hybrid expert model inference method includes the following steps: Step 401: Predict multiple candidate tokens based on the given token using a lightweight model.

[0116] Step 402: Predict the target expert model to be activated in each layer of the MoE model based on the token and each candidate token.

[0117] Step 403: Construct a sub-model corresponding to each candidate token based on the target expert model corresponding to each candidate token, and assign each candidate token and its corresponding sub-model to different computing nodes.

[0118] Step 404: Different computing nodes infer the pre-computation result of the next token in parallel based on the allocated candidate tokens and sub-models.

[0119] Step 405: When determining the actual token from multiple candidate tokens, use the corresponding pre-computed results.

[0120] The method provided by the above embodiments of this disclosure improves the efficiency and quality of model inference by fusing temporal inference sampling and spatial inference sampling and predicting multiple future tokens.

[0121] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a hybrid expert model reasoning device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0122] like Figure 5 As shown, the hybrid expert model inference device 500 of this embodiment includes: an acquisition unit 501, a determination unit 502, a construction unit 503, and an inference unit 504. The acquisition unit 501 is configured to acquire a hybrid expert model and an expert selection prediction model for the hybrid expert model, wherein the hybrid expert model includes multiple layers, each layer including multiple expert models; the determination unit 502 is configured to determine, based on a token and its context, a target expert model to be activated in different layers of the hybrid expert model using the expert selection prediction model; the construction unit 503 is configured to construct a sub-model based on the target expert model; and the inference unit 504 is configured to infer the context of the token based on the token and its context using the sub-model.

[0123] In this embodiment, the specific processing of the acquisition unit 501, determination unit 502, construction unit 503, and reasoning unit 504 of the hybrid expert model inference device 500 can be referred to Figure 2 The corresponding steps are 201, 202, 203, and 204 in the embodiment.

[0124] In some optional implementations of this embodiment, the device further includes a spatial reasoning unit (not shown in the figures), configured to: infer multiple candidate tokens based on the context of the token using a lightweight token predictor; for each candidate token, infer the context of the candidate token in parallel using a partially activated hybrid expert model based on the context of the token and the candidate token; verify the correct candidate token from the multiple candidate tokens using a fully activated hybrid expert model; and output the correct context of the candidate token.

[0125] In some optional implementations of this embodiment, the spatial reasoning unit is further configured to: determine candidate expert models to be activated in different layers of the hybrid expert model based on the context of the token and the candidate token through an expert selection prediction model; construct candidate sub-models based on the candidate expert models; assign the candidate token to the candidate sub-model, and have the candidate sub-model infer the context of the candidate token in parallel based on the context of the token and the candidate token.

[0126] In some optional implementations of this embodiment, the spatial reasoning unit is further configured to: infer the actual next token based on the context of the token using a fully activated hybrid expert model; match the next token with each candidate token, and take the successfully matched candidate token as the correct candidate token.

[0127] In some optional implementations of this embodiment, the spatial reasoning unit is further configured to: in response to a matching failure, infer and output the context of the next token based on the next token using a fully activated hybrid expert model.

[0128] In some optional implementations of this embodiment, the apparatus further includes a training unit (not shown in the figures), configured to: acquire information about the expert models activated during the process of reasoning about the following text of the target token sequence through a hybrid expert model based on the target token sequence; generate training samples based on the information and the target token sequence; and train an expert selection prediction model based on the training samples.

[0129] In some optional implementations of this embodiment, the construction unit 503 is further configured to: load the relevant parameters of the target expert model into memory; and construct a sub-model based on the relevant parameters in memory.

[0130] In some optional implementations of this embodiment, the determining unit 502 is further configured to: based on the token and the context of the token, output the activation probability of each expert model in each layer through expert selection prediction model; and determine the expert model in each layer whose activation probability exceeds a predetermined threshold as the target expert model to be activated.

[0131] In some optional implementations of this embodiment, the device further includes a parameter tuning unit (not shown in the figures) configured to dynamically adjust a predetermined threshold based on available computing resources and / or inference accuracy requirements.

[0132] In some optional implementations of this embodiment, the determining unit 502 is further configured to: based on the token and the context of the token, output the activation probability of each expert model in each layer by selecting an expert prediction model; and select a predetermined number of expert models with the highest activation probability in each layer as target expert models to be activated.

[0133] In some optional implementations of this embodiment, the apparatus further includes a parameter tuning unit (not shown in the figures) configured to adjust the number of candidate tokens based on at least one of the following: network throughput, computing resource utilization, and the amount of available computing resources.

[0134] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0136] An electronic device includes: one or more processors; and a storage device having one or more computer programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in process 200 or 400.

[0137] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in process 200 or 400.

[0138] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0140] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as road planning methods. For example, in some embodiments, the road planning method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the road planning method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the road planning method by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0147] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be servers in distributed systems or servers incorporating blockchain technology. Servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0148] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A hybrid expert model reasoning method, comprising: Obtain a hybrid expert model and an expert selection prediction model for the hybrid expert model, wherein the hybrid expert model comprises multiple layers, and each layer comprises multiple expert models; Based on the token and the context above it, the target expert model to be activated in different layers of the hybrid expert model is determined by the expert selection prediction model. Construct a sub-model based on the target expert model; Based on the token and its preceding context, the following context of the token is inferred through the sub-model.

2. The method according to claim 1, wherein, The method further includes: Based on the context of the token, multiple candidate tokens are inferred using a lightweight token predictor; For each candidate token, the following text of the candidate token is inferred in parallel using a partially activated hybrid expert model, based on the context of the token and the candidate token itself. The correct candidate token is verified from the multiple candidate tokens using a fully activated hybrid expert model; Output the following text for the correct candidate token.

3. The method according to claim 2, wherein, The step of inferring the context of the candidate token in parallel using a partially activated hybrid expert model, based on the context of the token and the candidate token, includes: Based on the context of the token and the candidate token, the candidate expert models to be activated in different layers of the hybrid expert model are determined by the expert selection prediction model. Construct candidate sub-models based on the aforementioned candidate expert models; The candidate token is assigned to the candidate sub-model, and the candidate sub-model infers the context of the candidate token in parallel based on the context of the token and the candidate token.

4. The method according to claim 2, wherein, The process of verifying the correct candidate token from the plurality of candidate tokens using a fully activated hybrid expert model includes: Based on the following text of the token, the actual next token is inferred through a fully activated hybrid expert model; The next token is matched with each candidate token, and the candidate token that matches successfully is taken as the correct candidate token.

5. The method according to claim 4, wherein, The method further includes: In response to a matching failure, the next token is inferred and output based on the next token using a fully activated hybrid expert model.

6. The method according to claim 1, wherein, The method further includes: Information about the expert models activated during the process of inferring the following text of the target token sequence through the hybrid expert model based on the target token sequence; Training samples are generated based on the information and the target token sequence; The expert selection prediction model is trained based on the training samples.

7. The method according to claim 1, wherein, The construction of sub-models based on the target expert model includes: Load the relevant parameters of the target expert model into memory; A sub-model is constructed based on the relevant parameters in memory.

8. The method according to claim 1, wherein, The process of determining the target expert model to be activated in different layers of the hybrid expert model based on the token and the context of the token, through the expert selection prediction model, includes: Based on the token and the context above it, the activation probability of each expert model in each layer is output through the expert selection prediction model. Expert models whose activation probability exceeds a predetermined threshold in each layer are identified as target expert models to be activated.

9. The method according to claim 8, wherein, The method further includes: The predetermined threshold is dynamically adjusted based on available computing resources and / or inference accuracy requirements.

10. The method according to claim 1, wherein, Based on the token and the context above the token, the expert selection prediction model determines the target expert model to be activated in different layers of the hybrid expert model, including: Based on the token and the context above it, the activation probability of each expert model in each layer is output through the expert selection prediction model. The expert models with the highest activation probability in each layer are selected as the target expert models to be activated.

11. The method according to claim 2, wherein, The method further includes: The number of candidate tokens is adjusted based on at least one of the following: Network throughput, computing resource utilization, and the amount of available computing resources.

12. A hybrid expert model inference apparatus, comprising: The acquisition unit is configured to acquire a hybrid expert model and an expert selection prediction model for the hybrid expert model, wherein the hybrid expert model includes multiple layers, each layer including multiple expert models; The determining unit is configured to determine the target expert model to be activated in different layers of the hybrid expert model based on the token and the context of the token, through the expert selection prediction model. The building unit is configured to build a sub-model based on the target expert model; The reasoning unit is configured to infer the context of the token from the sub-model based on the token and the context of the token.

13. An electronic device, comprising: One or more processors; Storage device, on which one or more computer programs are stored, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-11.

14. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images