A dynamic adaptive expert allocation method based on the sharpness of probability distribution

Abstract

The application provides a dynamic adaptive expert allocation method based on a probability distribution peak property, and relates to the technical field of expert allocation. The core of the method is to propose a peak adaptive routing mechanism, which analyzes the peak degree of the probability distribution output by the model, dynamically adjusts the expert selection threshold, and replaces the traditional fixed threshold or fixed number of experts routing strategy. Specifically, for each input data, the peak metric of its probability distribution is calculated, and an adaptive threshold is dynamically generated according to the metric value; an expert subset is selected from the expert set according to the threshold, and its normalized weight is calculated; finally, the output is calculated by weighting the selected expert subset. In the federated learning framework, the client only uploads the parameters of the selected experts to the server for aggregation, while the parameters of the unselected experts are retained locally for further training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of expert allocation, and particularly relates to a dynamic adaptive expert allocation method based on probability distribution peak property. BACKGROUND

[0002] Under the background of the rapid popularization of Internet of Things (IoT) devices and the surge in demand for edge computing, a large amount of data is generated at the network edge, which has important value in key fields such as intelligent medical treatment and automatic driving. However, the traditional centralized learning paradigm needs to gather data to the cloud for processing, which not only brings the risk of privacy leakage, but also produces high communication overhead. In order to solve the problems of privacy and communication, federated learning (FL) is proposed as a distributed learning framework, which realizes the protection of user privacy by training the model on the local device and uploading only the model parameters instead of the original data. However, with the continuous expansion of the model size (such as OpenELM-3B containing tens of billions of parameters), especially in the scene of limited resources of edge devices, it is difficult for the device to bear the heavy computing load, resulting in low training efficiency. Split learning (SL) as another technical path, by dividing the model into layers and only performing part of the calculation by the device, transfers the main calculation task to the server side, thereby reducing the burden of the device. However, with the increase in the number of participating devices, the intermediate layer activation values and gradients and other "broken data" transmitted between the device and the server increase sharply, forming a new communication bottleneck, which seriously affects the overall training efficiency (related research shows that the amount of broken data can account for more than 60% of the total communication cost).

[0003] Mixture of Experts (MoE) provides a new way to solve the above problems. By sparsely activating expert subnetworks, such as using a gating network to dynamically select a subset of experts, MoE can expand the model capacity while controlling the computational cost, making it particularly suitable for edge computing scenarios. However, traditional MoE routing methods, such as the Top-K strategy, have significant limitations. The fixed K value cannot adapt to the diversity of input data and the dynamic changes of device resources, leading to uneven expert load (expert utilization rate difference up to 20-40%), with some experts being overused while others are idle, and rigid allocation of computational resources (insufficient resources for difficult samples, waste of resources for simple samples), making it difficult to adjust flexibly according to the complexity of samples. To address these shortcomings, dynamic routing techniques have emerged, which can dynamically adjust the expert selection strategy based on the characteristics of input data (such as complexity or uncertainty), providing a new direction for the application of MoE in edge computing. Existing dynamic routing methods, such as Top-P (kernel sampling) or entropy-based threshold mechanisms, have improved the flexibility of routing to some extent, but still have drawbacks such as increased computational overhead, increased training complexity, uneven gradient propagation, and unresolved expert load imbalance.

[0004] Specifically, the core idea of Top-K routing is to select the top K experts with the highest probability from multiple expert candidates for activation and processing. This method optimizes computational efficiency and improves model scalability through sparse activation, making it particularly suitable for large-scale distributed training tasks. Dynamic expert quantity routing further extends this idea by dynamically adjusting the number of activated experts based on the characteristics of each input data, such as complexity and uncertainty. Unlike traditional MoE models that activate a fixed number of K experts (such as the Top-K strategy), dynamic routing can adaptively select the size of the expert subset based on input characteristics, such as Top-P (kernel sampling) or entropy-based threshold mechanisms, to achieve more flexible resource allocation. Top-K routing, such as GShard, uses a mechanism where each token selects the top K experts, which has significant advantages in computational efficiency and model scalability. However, it also has the following non-negligible drawbacks: load imbalance leads to overuse of some experts and idleness of others, computational resource waste due to fixed K value that cannot adapt to input complexity changes, hyperparameter tuning challenges in selecting K value that is sensitive to performance and requires extensive experimentation, and training instability due to inconsistent gradient propagation and convergence difficulties. Dynamic routing provides higher flexibility and computational efficiency by dynamically adjusting the number of experts using the Top-P mechanism, allowing automatic selection of activation paths based on input complexity and characteristics. However, its drawbacks include increased computational overhead due to dynamic threshold calculation and routing decisions, increased training complexity requiring additional optimization steps, uneven gradient propagation affecting model convergence speed, and unresolved expert load imbalance.

[0005] In view of this, the present application is proposed. SUMMARY

[0006] The application provides a dynamic adaptive expert allocation method based on the peak property of probability distribution, which can at least partially improve the above problems.

[0007] To achieve the above object, the application adopts the following technical scheme:

[0008] A dynamic adaptive expert allocation method based on the peak property of probability distribution, comprising:

[0009] Obtain input tokens, and sequentially perform probability distribution calculation, entropy calculation and normalization processing on the tokens to obtain probability distribution and normalized entropy;

[0010] Adopt a dynamic threshold adjustment mechanism to adjust the threshold according to a preset basic threshold and the normalized entropy;

[0011] Arrange the probability distribution in descending order, calculate the cumulative probability of each expert model, compare the cumulative probability with the adjusted threshold, and filter to generate an expert set, wherein the expert models in the expert set are trained expert models;

[0012] Upload the to-be-uploaded parameters of the expert set and the local data volume to the server, update the global model, and distribute the global expert learning general features of the updated global model to each expert model for training.

[0013] In summary, the present method aims to solve the technical bottlenecks of rigid resource allocation, huge communication overhead and uneven expert load of mixed expert models in distributed environments such as federated learning. The core of the present application lies in the proposed peak adaptive routing mechanism. This mechanism discards the traditional fixed K value or fixed threshold routing decision method, and innovatively uses the confidence information (i.e. the degree of "peak" or "flatness" of the distribution) contained in the model output probability distribution as the routing basis. Specifically, by calculating the information entropy and other peak metrics of the probability distribution, an adaptive threshold is dynamically generated based on the metrics, thereby realizing differentiated expert resource allocation for different complexity inputs. For complex samples with high uncertainty (flat distribution), the system automatically allocates more expert resources for processing; for simple samples with high confidence (sharp distribution), the number of experts used is reduced to save resources. This method upgrades the routing decision from static parameter dependence to dynamic response to the characteristics of the data itself, realizing the refinement and intelligentization of resource allocation.

[0014] Further, the application innovatively deeply integrates the spike adaptive routing mechanism in the federated learning framework. On the client side, based on the probability distribution of local data, a dynamic threshold is calculated and applied, and only part of the key expert's parameters are uploaded to the server, while the parameters of the unselected experts are retained locally for personalized training. This strategy creatively separates global model aggregation from local model personalization, and on the server side, it realizes the fusion and generalization of general knowledge through weighted aggregation (which can be further combined with dynamic weight adjustment based on confidence), while on the client side, it captures data-specific patterns using locally retained experts. This integrated solution not only significantly reduces the communication load between the client and the server through parameter screening, but also naturally enhances the model's adaptability to non-independent and identically distributed data, achieving a balance between personalized performance and generalization performance.

[0015] In addition, to improve the robustness and efficiency of the core mechanism, the application also introduces: 1) multi-scale adaptive threshold adjustment: a hierarchical threshold adjustment strategy is designed according to the characteristics of different network layers to enhance the adaptability to deep complex features; 2) regularization constraint optimization: smooth constraint term is used to prevent threshold from fluctuating sharply during training, improving training stability; 3) load balancing constraint: dynamically monitor and constrain the usage frequency of each expert, effectively alleviating the problem of uneven expert load; 4) adaptive compression transmission: according to the uncertainty of the data corresponding to the uploaded parameters, dynamically adjust the compression ratio to further reduce communication overhead while ensuring accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the dynamic adaptive expert allocation method based on the probability distribution spike property provided by the embodiments of the application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description will be made below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0018] REFERENCE Figure 1 As shown in the figure, the first embodiment of the application discloses a dynamic adaptive expert allocation method based on the probability distribution spike property, which can be executed by a dynamic adaptive expert allocation device based on the probability distribution spike property (hereinafter referred to as allocation device), in particular, by one or more processors in the allocation device to implement the following method:

[0019] S1, acquire the input token, and sequentially perform probability distribution calculation, entropy calculation and normalization processing on the token to obtain probability distribution and normalized entropy;

[0020] Specifically, step S1 further includes: obtaining the input token and calculating the probability distribution of the token using the Softmax function of the last layer of the model. ,in, R is the real number field, and K is the number of categories. Let i be the probability distribution;

[0021] According to probability distribution Calculate Shannon entropy Furthermore, Shannon entropy is used as a measure of peak size, where a larger entropy value indicates a flatter distribution (higher uncertainty), and a smaller entropy value indicates a sharper distribution (higher confidence).

[0022] Shannon entropy Normalize to a preset interval (e.g., [0,1]) to facilitate threshold adjustment and obtain the normalized entropy. ,in, Indicates a high level of confidence. This indicates high uncertainty (the distribution is close to a uniform distribution).

[0023] In this embodiment, for each input token, a category probability distribution is first output. The peak metric of this distribution is then calculated to quantify model uncertainty. The peak metric is a core basis for routing decisions, and its accuracy directly affects resource allocation efficiency. This invention introduces a quantum entropy enhancement mechanism based on traditional Shannon entropy to improve the robustness and expressive power of uncertainty quantification.

[0024] Specifically, first, the input data to be processed is acquired, which is typically represented in the form of tokens in tasks such as natural language processing. For each input token, the model first performs forward computation up to the last layer and generates a class probability distribution vector using the Softmax function. Specifically, assuming the original logistic value of the model output is , the probability distribution calculated by the Softmax function is then used. This probability distribution intuitively reflects the model's confidence in assigning the current input token to each expert (or class).

[0025] Secondly, to quantify the concentration of this probability distribution (i.e., its "peaked" property), this invention uses Shannon entropy from information theory as a measure of peak value. The Shannon entropy of this probability distribution is calculated. The magnitude of the Shannon entropy directly characterizes the uncertainty of the distribution: when the probability is highly concentrated on one or a few experts (the distribution is "peaked"), the entropy value is small, indicating that the model has a high degree of confidence in its judgment of the input; conversely, when the probability is evenly distributed across all experts (the distribution is "flat"), the entropy value reaches its maximum, indicating that the model's judgment of the input has high uncertainty.

[0026] In addition, for the subsequent dynamic threshold adjustment mechanism, the calculated Shannon entropy is normalized to a fixed interval, for example, [0, 1]. A feasible normalization method is to scale it with the theoretical maximum entropy. The technical effect of this step is that it converts the confidence information implicitly in the model into a standardized and interpretable scalar index, providing accurate and uniform quantitative basis for subsequent adaptive decision-making closely related to input characteristics.

[0027] S2, adopting a dynamic threshold adjustment mechanism, adjusting the threshold according to the preset basic threshold and the normalized entropy;

[0028] Specifically, step S2 further comprises: adopting a dynamic threshold adjustment mechanism, adjusting the threshold according to the preset basic threshold and the normalized entropy, specifically: setting the basic threshold , and adjusting the threshold according to the basic threshold and the normalized entropy , and the adjustment formula is: , is a hyperparameter (default 0.2) that controls the strength of the spike influence. Among them, the threshold is increased when the entropy is high (i.e. ), and more experts are included; low entropy indicates high confidence, and the threshold is reduced to save resources.

[0029] It also includes: to solve the limitations of a single scale, a multi-scale threshold adjustment mechanism is introduced, a hierarchical threshold strategy is designed based on network hierarchy and data complexity, and the formula is: , is a multi-scale fusion threshold, which is the final routing threshold calculated by integrating information of each network layer. This threshold replaces the traditional single global threshold and can adapt to the complexity of features of different network depths. L is the number of network layers, is the layer weight, is the layer-related scaling factor, and deep networks are given a larger to enhance adaptability, and shallow networks use a smaller to maintain stability, is the lth normalized entropy;

[0030] To prevent excessive fluctuations in the threshold, a regularization constraint term is added, and the formula is: , is the final threshold after regularization. This parameter is the actual threshold used for expert selection after smoothing, which not only retains the adaptability of the dynamic threshold P dynamic to input characteristics, but also suppresses abnormal fluctuations through regularization constraints. This threshold ensures the stability of the training process while maintaining adaptability to different input samples. is a hyperbolic tangent function, which plays a role of smoothing the boundary in this constraint term, is a regularization strength, is a smoothing factor, is a historical threshold mean value. This constraint ensures the smooth change of the threshold value, avoiding the impact of sharp fluctuations on the stability of training.

[0031] In the embodiment, a dynamic threshold adjustment mechanism is used to replace the traditional fixed threshold strategy, and the number of expert selections is optimized in real time through the spike metric. The mechanism adjusts the threshold value adaptively according to the uncertainty of the input data, and realizes fine-grained resource allocation. The application introduces two innovations, multi-scale adaptive adjustment and regularization constraint, on the basis of the dynamic threshold framework, to improve the stability and adaptability of threshold adjustment.

[0032] Specifically, the goal of this step is to generate an adaptive expert selection threshold closely related to the current input characteristics, completely replacing the strategy of rigid fixed K value or fixed probability threshold P in the traditional hybrid expert model. A basic threshold is preset. This value serves as a reference point for adjustment, for example, it can be set to 0.4. Then, according to the normalized entropy calculated in step S1, the final threshold of the current token is dynamically generated through a linear adjustment formula. The working principle and beneficial effects of this mechanism are very significant: when the uncertainty of the input sample is high, the threshold value increases accordingly. This means that the system will set a higher cumulative probability threshold, thereby automatically including more experts to cooperatively process this "difficult sample", allocating more abundant model capacity and computing resources to it to improve processing accuracy. Conversely, for "simple samples" with high model confidence, the threshold value decreases, and the system will select a small number of key experts under more stringent conditions, avoiding resource waste. This process realizes a fundamental change from "static resource allocation" to "dynamic on-demand allocation", and its direct beneficial effects are: at the model level, it realizes intelligent scheduling of computing resources, so that limited computing power is always prioritized for the most complex input processing, thereby significantly improving the overall computing efficiency and resource utilization of training and inference.

[0033] To further enhance the robustness and flexibility of this adaptive mechanism in different network depths and complex scenarios, the application introduces two innovative extension mechanisms. The first is a multi-scale threshold adjustment mechanism. Considering that the abstract level and complexity of the features extracted by different levels of deep neural networks are different, a single-scale threshold adjustment may not be globally optimal. Therefore, the application designs a hierarchical threshold strategy based on the network hierarchy. Specifically, for the l-th layer network, its dynamic threshold is independently adjusted by the normalized entropy of the layer. The beneficial effects of this mechanism are to realize hierarchical fine-grained control, so that the routing decision can match the hierarchical feature extraction process of the network itself, thereby maintaining efficient and stable resource allocation in complex model architectures.

[0034] The second is a regularization constraint optimization. In the training process, the dynamic threshold value completely dependent on the instantaneous input calculation can appear violent fluctuations, which is not conducive to the stable convergence of the model training. In order to prevent the threshold from excessive fluctuation, the present application introduces a regularization constraint term in the threshold calculation, which smoothes the threshold used finally. The beneficial effect of this innovation is to significantly improve the stability of the training process. It effectively avoids the dramatic change of the expert selection strategy caused by the sudden change of the characteristics of a single batch of data by introducing the inertia constraint of historical information, ensures the smooth evolution of the routing decision, thereby accelerating the model convergence and improving the robustness of the final model.

[0035] S3, arranging the probability distribution in descending order, and calculating the cumulative probability of each expert model, comparing the cumulative probability with the adjusted threshold, and screening to generate an expert set, wherein the expert models in the expert set are trained expert models;

[0036] Specifically, step S3 further comprises: arranging the probability distribution in descending order, and the formula is: , k = 1, 2,..., K, to obtain ordered probability values and corresponding expert indexes I;

[0037] Calculate the cumulative probability , and the formula is: , and screen out expert models to generate an expert set , t is the number of selected expert models, is the tth cumulative probability;

[0038] Calculate the normalized weight of the expert model in the expert set , wherein the weight of the unselected expert model is 0;

[0039] According to the normalized weight and the expert set, the expert selection result is obtained, is the expert model with the ith largest probability, i = 1, 2,..., t.

[0040] Further comprising: to alleviate the problem of uneven use of experts, a load balancing constraint is introduced, and the selection strategy is dynamically adjusted by a regularization term, and the formula is: , wherein, is a load difference threshold (default 0.1), is an expert use frequency vector (total number of experts), and N is the total number of experts in the expert network. This parameter determines the capacity and complexity of the MoE model, ​The optimal number of experts. The minimum number of experts calculated through optimization, satisfying both cumulative probability coverage and load balancing constraints. This parameter ensures that load balancing among experts is achieved while meeting the probability coverage requirement. t The value range is [1, N], where N is the total number of experts. This constraint ensures that the usage frequency differences of the selected expert subset are controllable, avoiding overload of some experts. Load information is updated in real time through a sliding window, enhancing system robustness.

[0041] In this embodiment, experts are sorted according to their probability distribution, cumulative probabilities are calculated, a set of experts is selected, and normalized weights are calculated for them. The output of the expert model is then obtained. A load balancing constraint is introduced to improve routing efficiency and stability. Steps S1-S3 are executed by the client to achieve dynamic expert allocation.

[0042] Specifically, the core task of this step is to use the adaptive threshold generated in step S2 to select the most relevant subset from all experts, assign computational weights to it, and finally complete the sparsity calculation. This process completely changes the traditional Top-K fixed-number selection mode, realizing dynamic determination of the number of experts based on model confidence and sample complexity. The original probability distribution obtained in step S1 is sorted in descending order of probability value. At the same time, the corresponding expert index sequence is recorded. This sorting operation ensures that the experts with the highest probability are placed at the beginning of the sequence, preparing for subsequent cumulative selection.

[0043] Next, calculate the cumulative sum of the probabilities after sorting. Define the cumulative probability C of the first j experts. j The cumulative probability is compared with the final dynamic threshold. The expert-chosen decision is to find the minimum index t such that... The selection is now complete. At this point, the chosen expert set S is selected. The direct benefit of this mechanism is the complete adaptability of the number of experts: for complex samples requiring a high threshold, the cumulative probability requires adding more experts to reach the threshold, resulting in a larger t-value and activating more experts; for simple samples, only a few high-probability experts are needed to meet a lower threshold, resulting in a smaller t-value. This ensures the "on-demand" nature of computational resource allocation, which is key to improving overall computational efficiency.

[0044] After determining the expert set, each expert needs to be assigned a normalized weight for weighting the output. The weights are calculated by renormalizing only the original probabilities of the selected experts. For experts not included in set S, their weights are set to 0. Finally, the output of the MoE layer is obtained by weighted summation of the outputs of the selected experts. The advantage of this weighting scheme is that it not only sparsifies the expert selection but also precisely calibrates the contribution, ensuring that the output is mainly determined by a group of highly relevant experts whose total confidence exceeds an adaptive threshold, thus balancing efficiency and representation accuracy.

[0045] To address the issue of unbalanced expert load that may arise during long-term training (i.e., some experts are frequently selected while others are ignored), this invention innovatively introduces a load balancing constraint in routing decisions. An expert usage frequency vector is defined. During routing decisions, a regularization term is introduced to dynamically influence the selection strategy. This can take the form of adjusting the probability distribution or influencing threshold judgments, aiming to encourage the selection of experts with less recent usage. An exemplary constraint objective is to control the difference in expert usage frequency among the selected expert subset S. The frequency vector F is updated in real-time through a sliding window mechanism to reflect recent expert usage. The core benefit of introducing the load balancing constraint is a significant improvement in the long-term stability and resource utilization of the distributed training system. It effectively avoids the problem of a few experts being overloaded while most experts are idle due to the "Matthew effect," ensuring that all experts are fully trained and utilized, thereby effectively maximizing the overall model capacity and reducing the performance bottleneck or failure risk that may result from the overload of a single expert.

[0046] S4 uploads the parameters to be uploaded from the expert set and the local data volume to the server, updates the global model, and distributes the global expert learning general features of the updated global model to each expert model for training. Unselected expert parameters are retained locally and not uploaded, thus reducing communication volume by 50%.

[0047] Specifically, step S4 further includes: introducing an entropy-based adaptive compression mechanism to quantize and encode the parameters of the expert models in the expert set S to obtain the parameters to be uploaded. The compression ratio is dynamically adjusted: high-entropy samples (high uncertainty) use a low compression ratio to preserve details, while low-entropy samples use a high compression ratio to save bandwidth. The adjustment formula is: , For compression ratio, This is the maximum compression factor (default 0.8). This mechanism reduces communication overhead by 15-20% while maintaining accuracy.

[0048] Parameters to be uploaded and local data volume Uploaded to the server, the server controls the upload based on the parameters to be uploaded. and local data volume A weighted aggregation process is performed to update the global model, specifically as follows:

[0049] Calculate global parameters k=1,...,K, j=1,...,K;

[0050] Weights are based on the proportion of data volume from each client to ensure fairness. A data quality-aware weight optimization mechanism is introduced, adjusting the aggregated weights according to the confidence level (e.g., entropy value) of the local expert model on each client. The adjustment assigns higher weight to low-entropy clients (high confidence), using the following formula: , For the k-th normalized entropy, Let be the j-th normalized entropy. This strategy improves the accuracy of the aggregation model, especially in non-independent identically distributed (Non-IID) data scenarios.

[0051] The updated global expert-learned general features are distributed to each expert model for training. The aggregated global expert-learned general features promote model generalization. Meanwhile, expert parameters not uploaded continue training locally on the client side, capturing client-specific patterns and enhancing personalized performance. Local experts use regularization terms to prevent overfitting.

[0052] In this embodiment, in a Federated Learning (FL) scenario, each trained client uploads its local expert parameters to the server (untrained clients do not upload). The server then performs weighted aggregation of the parameters uploaded by each client to update the global model. The updated global model can then be used by each client for learning. This invention introduces an innovative adaptive compressed transmission mechanism to optimize the efficiency and adaptability of federated learning. The integration process and innovations are detailed below.

[0053] Specifically, after local training is completed on the client side, not all expert parameters are uploaded to the server. Instead, the client only uploads the parameters of a selected subset along with its local data volume to the central server. Crucially, expert parameters not in the subset are completely preserved on the client's local machine and are not included in this upload. This selective upload strategy is one of the key innovations of this invention, and its direct and significant benefit lies in the substantial reduction of communication load by transmitting only the activated expert parameters that are most representative of the current client data. Experiments show that compared to the traditional approach of uploading all expert parameters, this invention can reduce the total number of parameters uploaded in a single session by approximately 50% on average, effectively alleviating the communication bottleneck of federated learning in bandwidth-constrained edge environments.

[0054] To further explore the potential for communication optimization, this invention introduces an entropy-based adaptive compression mechanism before parameter uploading. This mechanism quantizes and encodes the parameters of the selected expert set. Its core innovation lies in the fact that the compression rate is not fixed, but dynamically adjusted based on the overall uncertainty of the input sample that triggered the expert selection (i.e., its normalized entropy). The beneficial effect of this adaptive compression mechanism is that it achieves an intelligent trade-off between communication efficiency and knowledge fidelity. Without significantly affecting model performance, it can further reduce communication overhead by 15% to 20% on top of the aforementioned selective uploading, resulting in a cumulative and amplified overall communication optimization effect.

[0055] After collecting compressed expert parameters and their data volume information uploaded by all clients, the server performs weighted aggregation to update the global model. The global parameter update follows the basic idea of ​​federated averaging, but the weight allocation strategy is enhanced. First, the basic aggregation weights are calculated based on the proportion of each client's local data volume to ensure basic fairness. Specifically, for any expert, their global parameters are obtained by a weighted average of the contributions from all clients who uploaded those expert parameters. To further improve the quality of the aggregated global model, especially in the face of challenges with non-independent and identically distributed data, this invention introduces a data quality-aware weight optimization mechanism. During aggregation, the server considers not only the data volume but also the confidence level of the client's local model (or its selected expert). The average confidence level of the client's local training can be indirectly reflected by the average normalized entropy of its data. The aggregation weights are adjusted. This formula assigns higher aggregation weights to clients with low average entropy (i.e., high average confidence). Its beneficial effects are: guiding the global model to learn more from clients with high-quality data and better local model training, thereby effectively filtering out noisy updates caused by poor data quality or insufficient local training, significantly improving the aggregation accuracy and convergence speed of the global model, and this advantage is particularly prominent in non-independent and identically distributed scenarios.

[0056] After aggregation, the server distributes the updated global expert parameters to all clients. Clients use these parameters to update their local experts. These global experts learn common features and patterns extracted from each client, enhancing the model's generalization ability. Simultaneously, a parallel and crucial process is the continuous personalized training of local experts. Expert parameters not selected in step S3 and therefore not uploaded to the server continue to be trained locally on the client using their private data. These local experts are specifically dedicated to capturing the data distribution and patterns unique to that client. To prevent these local experts from overfitting their local data and deviating too far from general knowledge, regularization constraints are typically applied (e.g., constraining their parameters not to deviate too far from the parameters of the corresponding global experts). This architecture of "global-local" expert separation and collaborative evolution is another core beneficial effect of this invention. It cleverly achieves a balance between "generalization" and "personalization" in federated learning: global experts ensure the model's general performance, while local experts deeply adapt to individual needs, enabling the model ultimately deployed on edge devices to simultaneously possess powerful general task processing capabilities and excellent local personalized performance.

[0057] Specifically, in this embodiment, the method primarily addresses the core bottlenecks of hybrid expert models in complex application scenarios (such as federated learning), including rigid resource allocation, low communication efficiency, and insufficient utilization of model uncertainty. Specifically, PAR overcomes traditional limitations in the following ways: First, it dynamically adjusts the expert selection threshold using the shape characteristics of the probability distribution (such as peaks and flatness), where the peak level is quantified through information entropy or variance to reflect model confidence. Second, it employs an adaptive mechanism to replace static parameters (such as fixed K values ​​or thresholds), thereby optimizing expert allocation in real time based on instance-level differences in input data (such as complexity variations) and the dynamics of the edge environment (such as network conditions). Compared to traditional MoE routing methods (such as Top-K or fixed threshold strategies), PAR avoids the limitations of relying solely on probability values ​​for decision-making, fully leveraging the uncertainty information of the probability distribution to improve routing flexibility and efficiency. This mechanism provides more refined resource control for scenarios such as federated learning, effectively solving the problems of load imbalance and communication overhead.

[0058] In summary, the dynamic adaptive expert allocation method based on the peak properties of probability distributions is an adaptive MoE routing method (Peak-Adaptive Routing, PAR) based on the peak properties of probability distributions. It is an innovative routing mechanism whose core lies in replacing traditional fixed threshold strategies with dynamic analysis of the peak degree of probability distributions (such as entropy, variance, and other statistical characteristics) to achieve intelligent expert selection. Specifically, Peak-Adaptive Routing (PAR) is an adaptive MoE routing method based on the peak properties of probability distributions, aiming to replace traditional fixed threshold strategies with dynamic analysis of the peak degree of probability distributions (such as entropy and variance) to achieve intelligent expert selection. The core idea of ​​PAR is to dynamically adjust the expert selection threshold using the peak properties of probability distributions (such as the uncertainty reflected by entropy). Specifically, for each input token, PAR calculates the peak metric of its probability distribution (such as entropy), dynamically adjusts the threshold based on this metric, and then selects a subset of experts whose cumulative probability exceeds the threshold. In this method, the overall process is divided into four stages: peak metric calculation, dynamic threshold adjustment, expert selection and routing, and federated learning integration; this method improves resource allocation efficiency and communication efficiency while ensuring coverage.

[0059] This method's Mixture of Experts (MoE) model significantly expands model capacity while effectively controlling computational costs by sparsely activating expert subnetworks. Traditional routing methods, such as Top-K selection or fixed-threshold routing, while ensuring basic coverage, ignore the local differences in probability distribution characteristics, leading to a lack of flexibility in resource allocation. This invention innovatively utilizes the peak properties of probability distribution to improve the expert selection mechanism, proposing the Peak-Adaptive Routing (PAR) method. This method dynamically analyzes the peak degree of probability distribution (such as information entropy, variance, and other statistical characteristics) to adaptively adjust the threshold, replacing the traditional fixed threshold P, and achieving more refined expert selection. Specifically, PAR dynamically calculates the activation threshold based on the probability distribution characteristics of each input instance, enabling complex samples to automatically obtain more expert resources and reducing resource allocation for simple samples. Furthermore, PAR is innovatively applied to federated learning scenarios: the client calculates the expert selection strategy based on local data distribution, only uploads key expert parameters to the server for aggregation, and retains unselected experts for local processing. This mechanism not only significantly reduces communication overhead (estimated to be reduced by more than 50%), but also enhances the personalization capabilities of the model by retaining local experts, providing an efficient distributed learning solution for resource-constrained edge computing environments.

[0060] The Peak-Adaptive Routing (PAR) proposed in this invention is an adaptive MoE routing method based on the peak property of probability distribution. It adjusts the expert selection threshold by dynamically analyzing the peak degree (such as entropy or variance) of the probability distribution, replacing the traditional fixed threshold strategy. Extensive experimental verification shows that compared with traditional methods, this invention exhibits the following significant benefits in multiple dimensions: 1. PAR significantly improves the model's performance in downstream tasks by dynamically adapting to input complexity. Specific effects include higher accuracy. PAR leverages the advantage of dynamic expert allocation, further optimizing resource allocation through peak properties, allowing harder samples to receive more expert support, thereby improving overall accuracy. 2. PAR significantly reduces computational overhead through intelligent resource allocation, specifically in terms of improved computational efficiency. Dynamic routing converges more than twice as fast as Top-2 routing due to more rational resource allocation. PAR reduces redundant computation through entropy regularization, improving training and inference speed. 3. In federated learning (FL) scenarios, PAR significantly reduces communication overhead while maintaining model performance. PAR can reduce the amount of parameter transmission. PAR selects a subset of experts through peak selection, uploading only active parameters and retaining unselected experts locally, reducing the average upload rate to 30-50% and alleviating bandwidth bottlenecks. Furthermore, PAR enhances personalization; within FL, PAR allows local retention of expert captures of client-specific patterns, while global experts learn general knowledge.

[0061] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A dynamic adaptive expert allocation method based on the peak property of probability distribution, characterized in that, include: Obtain the input token, and sequentially perform probability distribution calculation, entropy calculation, and normalization processing on the token to obtain the probability distribution and normalized entropy; A dynamic threshold adjustment mechanism is adopted to adjust the threshold based on the preset base threshold and normalized entropy; The probability distributions are arranged in descending order, and the cumulative probability of each expert model is calculated. The cumulative probability is compared with the adjusted threshold, and an expert set is generated by filtering. The expert models in the expert set are pre-trained expert models. The parameters to be uploaded from the expert set and the amount of local data are uploaded to the server to update the global model. The global expert learning general features of the updated global model are then distributed to each expert model for training. A multi-scale threshold adjustment mechanism is introduced. Based on the network hierarchy and data complexity, a hierarchical threshold strategy is designed, with the following formula: , This represents a multi-scale dynamic threshold, where L is the number of network layers. For layer weights, This is the layer-dependent scaling factor. For the l-th normalized entropy, Basic threshold; Add a regularization constraint term; the formula is as follows: , This is the final threshold after regularization. It is the hyperbolic tangent function. For regularization strength, As a smoothing factor, This is the historical average threshold. The threshold value is used.

2. The dynamic adaptive expert allocation method based on the peak property of probability distribution according to claim 1, characterized in that, The input token is obtained, and its probability distribution, entropy, and normalization processes are performed sequentially to obtain the probability distribution and normalized entropy, specifically as follows: Obtain the input token and calculate its probability distribution using the Softmax function. ,in, R is the real number field, and K is the number of categories. Let i be the probability distribution; According to probability distribution Calculate Shannon entropy Furthermore, Shannon entropy is used as a measure of peak size, where a larger entropy value indicates a flatter distribution, and a smaller entropy value indicates a sharper distribution. Shannon entropy Normalize to a preset interval to obtain the normalized entropy. ,in, Indicates a high level of confidence. This indicates high uncertainty.

3. The dynamic adaptive expert allocation method based on the peak property of probability distribution according to claim 2, characterized in that, A dynamic threshold adjustment mechanism is adopted, which adjusts the threshold based on a preset base threshold and normalized entropy. Specifically, a base threshold is set. And based on the basic threshold and normalized entropy For threshold Adjustments are made, and the adjustment formula is as follows: , This is a hyperparameter.

4. The dynamic adaptive expert allocation method based on the peak property of probability distribution according to claim 3, characterized in that, The probability distributions are arranged in descending order, and the cumulative probability of each expert model is calculated. The cumulative probability is compared with an adjusted threshold, and an expert set is generated through selection. Specifically: The probability distributions are sorted in descending order. The formula for arranging is: Given k=1,2,...,K, we obtain ordered probability values ​​and corresponding expert indexes I; Calculate cumulative probability Its formula is: And filter out The expert model generates an expert set. t is the number of expert models selected. Let t be the cumulative probability; Calculate the normalized weights of the expert models in the expert ensemble. The weight of the unselected expert models is 0. Based on normalized weights By gathering experts, we can obtain the expert selection results. , Let i be the expert model with the highest probability, i=1,2,...,t.

5. The dynamic adaptive expert allocation method based on the peak property of probability distribution according to claim 4, characterized in that, Also includes: introduction Load balancing constraints, through dynamic adjustment of the selection strategy using regularization terms, have the following formula: ,in, The load difference threshold, The frequency vector is used for experts, where N is the total number of experts in the expert network. This represents the optimal number of experts.

6. The dynamic adaptive expert allocation method based on the peak property of probability distribution according to claim 4, characterized in that, The parameters to be uploaded from the expert set and the amount of local data are uploaded to the server to update the global model. The updated global model's global expert learning features are then distributed to each expert model for training. Specifically: An entropy-based adaptive compression mechanism is introduced to quantize and encode the parameters of the expert models in the expert set S, thereby obtaining the parameters to be uploaded. The compression ratio is dynamically adjusted using the following formula: , For compression ratio, The maximum compression factor; Parameters to be uploaded and local data volume Uploaded to the server, the server controls the upload based on the parameters to be uploaded. and local data volume Perform weighted aggregation to update the global model; The updated global expert learning general features of the global model are distributed to each expert model for training.

7. The dynamic adaptive expert allocation method based on the peak property of probability distribution according to claim 6, characterized in that, Parameters to be uploaded and local data volume Uploaded to the server, the server controls the upload based on the parameters to be uploaded. and local data volume A weighted aggregation process is performed to update the global model, specifically as follows: Calculate global parameters k=1,...,K, j=1,...,K; A data quality-aware weight optimization mechanism is introduced, which adjusts the aggregate weights based on the confidence level of the local expert model on the client side. The adjustment is made using the following formula: , For the k-th normalized entropy, Let be the j-th normalized entropy.

Images