Optimization methods and electronic devices for model training

Abstract

This application provides an optimization method and electronic device for model training. The method involves acquiring training data; inputting the training data into a hybrid expert model and obtaining expert routing information for each hybrid expert layer in at least one hybrid expert layer of the model; analyzing the routing scheduling of at least one expert network in each hybrid expert layer based on the expert routing information to determine the routing indices of each hybrid expert layer; performing structural adjustments on target hybrid expert layers whose routing indices meet structural adjustment conditions to obtain at least one updated hybrid expert layer; and training the hybrid expert model based on at least one updated hybrid expert layer to obtain the trained hybrid expert model. This reduces the computational and communication overhead caused by unnecessary structural adjustments during model training, effectively improving the utilization of processor computing resources.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to an optimization method and electronic device for model training. Background Technology

[0002] Currently, various processors (such as Graphics Processing Units, GPUs) mostly employ Mixture-of-Experts (MoE) models when processing tasks. This model, by deploying multiple parallel expert networks at each layer and using routers to select a small number of experts to participate in the computation, significantly reduces actual computational overhead while maintaining a large parameter scale. Therefore, it is widely used in fields such as natural language processing, computer vision, and multimodal tasks. However, in practical applications, these processors suffer from low computational resource utilization. This is because when adjusting the structure of the Mixture-of-Experts model, these processors often rely on fixed periods or static rules, which may lead to wasted computational and communication overhead. Summary of the Invention

[0003] This application provides an optimization method, apparatus, electronic device, and storage medium for model training. Its main purpose is to address the problem of low processor computing resource utilization caused by hybrid expert models triggering structural adjustments based on fixed periods or static rules in related technologies.

[0004] According to a first aspect of this application, an optimization method for model training is provided, comprising: Obtain the training data; The training data is input into the hybrid expert model, and the expert routing information of each hybrid expert layer in at least one hybrid expert layer in the hybrid expert model is obtained. Based on expert routing information, the routing scheduling of at least one expert network in each hybrid expert layer is analyzed to determine the routing metrics of each hybrid expert layer. The routing scheduling includes at least one of the following: the load balancing status of at least one expert network, the effective expert ratio, and the routing change status. The target hybrid expert layer whose routing metrics meet the structural adjustment conditions is structurally adjusted to obtain at least one updated hybrid expert layer. The hybrid expert model is then trained based on the at least one updated hybrid expert layer to obtain the trained hybrid expert model.

[0005] According to a second aspect of this application, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the optimization method for model training described in the first aspect above.

[0006] This application provides an optimization method, electronic device, and storage medium for model training, relating to the field of model training technology. Compared with related technologies, this application obtains training data; inputs the training data into a hybrid expert model and obtains expert routing information for each hybrid expert layer in at least one hybrid expert layer of the hybrid expert model; based on the expert routing information, analyzes the routing scheduling of at least one expert network in each hybrid expert layer to determine the routing indicators of each hybrid expert layer. The routing scheduling includes at least one of the following: load balancing status of at least one expert network, effective expert ratio, and routing change status; and performs structural adjustments on the target hybrid expert layer whose routing indicators meet the structural adjustment conditions. The process involves adjusting the model to obtain at least one updated hybrid expert layer, and then training the hybrid expert model based on this updated layer to obtain the trained hybrid expert model. This approach enables precise triggering of structural adjustment operations based on the routing and scheduling of each expert layer during the hybrid expert model training process. This effectively avoids the blindness of adjustments triggered by fixed periods or static rules in related technologies. It reduces unnecessary computational and communication overhead caused by structural adjustments and allows for timely intervention when expert loads are unbalanced or resource utilization is inefficient. This significantly improves the resource utilization balance and overall operating efficiency of the hybrid expert model, while also enhancing the stability of the model training process, thereby effectively improving the utilization rate of processor computing resources.

[0007] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0008] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein: Figure 1 A flowchart illustrating the first model training optimization method provided in this application embodiment; Figure 2 A flowchart illustrating the second model training optimization method provided in this application embodiment; Figure 3 A flowchart illustrating the third model training optimization method provided in this application embodiment; Figure 4 A flowchart illustrating the fourth model training optimization method provided in this application embodiment; Figure 5 A flowchart illustrating the fifth model training optimization method provided in this application embodiment; Figure 6 This application provides a schematic diagram of a specific hybrid expert model training system. Figure 7 A schematic diagram illustrating a specific optimization of model training provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an optimization device for model training provided in an embodiment of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0010] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0011] As the scale of deep learning models continues to expand, the number of model parameters and computational costs are growing exponentially. The bottlenecks of traditional dense neural network structures in terms of computational efficiency and training cost control are becoming increasingly prominent. Mixture-of-Experts (MoE) models, with their sparse activation design, have become a core technology for large-scale model training. This model introduces multiple expert networks into the network and uses gating networks to dynamically route input samples or tokens. During training, only a portion of the expert networks are activated to participate in computation, which can significantly reduce the computational cost per sample while maintaining high model capacity. In the operational logic of the hybrid expert model, the expert routing mechanism relies on the probability distribution of the gating network output to allocate input samples or tokens, thereby achieving load balancing and functional division among experts. However, in actual large-scale training, due to multiple factors such as uneven data distribution, differences in the convergence characteristics of gating networks, and different training stages, many practical problems are still exposed: some expert sub-networks are frequently selected and bear a large amount of computational load, while others are underutilized or even idle for a long time; the functions of expert sub-networks gradually converge, resulting in increased model redundancy; the expert load distribution changes dynamically with the training process, and there are significant differences in the expert load characteristics of different network layers.

[0012] To address the aforementioned issues, related technologies improve expert utilization and reduce system overhead through expert pruning, expert reordering, and load balancing constraints. Among these, expert reordering technology can reduce cross-device communication overhead by adjusting the organization of expert subnetworks in computing units, and can also be combined with expert pruning strategies to further compress model size. However, existing expert reordering technologies still have significant shortcomings in triggering mechanisms and execution strategies, making it difficult to ensure both model training stability and system operating efficiency.

[0013] Currently, structural adjustment schemes for hybrid expert models in related technologies are mainly divided into two categories. One category is a single-time or periodic scheduling-based expert network structural adjustment scheme, which performs a rearrangement operation once or periodically according to a preset number of training steps or rounds during the training process, and prioritizes the deployment of high-load expert networks in low-communication-cost computing units based on the number of samples received by experts or the activation frequency. The other category is a structural adjustment scheme based on a single statistical indicator, which uses routing frequency or load size as the sole metric to prune low-load expert networks and rearrange the remaining expert networks to adapt to the computing resource allocation.

[0014] While these two types of structural adjustment schemes attempt to address the issues of load imbalance and model redundancy to some extent, they still have several key shortcomings, as follows: 1) Lack of adaptive triggering mechanism, unreasonable timing of structural adjustment, reliance on fixed period or static rules to trigger rearrangement, inability to adjust in real time according to the dynamic changes of expert network routing, invalid structural adjustment when routing distribution is stable will cause waste of computing and communication overhead, and inability to intervene in time when load is unbalanced or routing degrades. 2) Decision-making driven by a single indicator is difficult to fully reflect the state of the expert network. Decisions made based solely on single indicators such as routing frequency and load size do not take into account multiple dimensions such as expert utilization, routing distribution entropy, number of effective experts, and routing change trends. This can easily lead to bias in the assessment of expert importance and affect the overall performance of the model. 3) Frequent reordering leads to training instability. In the early stages of training or when the gating network has not fully converged, periodic reordering can introduce strong structural nonstationarity, which can disrupt the cooperative relationship between experts and the gating network, causing training oscillations or even performance degradation. 4) The differences between the training phase and network layers were not fully considered. Differentiated constraints were not set for different phases such as warm-up and stable training, and the differences in load characteristics of different network layers were not taken into account. This resulted in the reordering being performed at an inappropriate time or level, affecting the model's convergence efficiency. 5) There is still considerable room for improvement in the efficiency of system resource utilization. In distributed training scenarios such as multi-machine and multi-card, unnecessary structural adjustments will significantly increase the cost of cross-device communication and synchronization, reduce the overall training throughput, and result in inefficient utilization of computing resources.

[0015] To address the problems existing in related solutions, this application proposes an optimization method for model training. Based on the Layer-wise Adaptive Expert Pruning (LAEP) method in the training process of hybrid expert models, this method determines routing indicators based on the dynamic expert routing scheduling of each hybrid expert layer in the hybrid expert model. Targeted structural adjustment operations are performed only on target hybrid expert layers whose routing indicators meet the preset structural adjustment trigger conditions. This effectively improves the trigger accuracy of structural adjustments for each hybrid expert layer, reduces unnecessary structural adjustment operations, and enhances the resource utilization balance and training stability of the hybrid expert model, significantly optimizing model training efficiency in large-scale distributed training scenarios.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Figure 1 This is a flowchart illustrating the first model training optimization method provided in the embodiments of this application.

[0018] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the training data.

[0019] In some embodiments, the training data refers to the original task sample data input to the hybrid expert model for completing iterative model training and routing behavior statistics. It is the basic data that the model can use to perform lexical parsing and hierarchical forward operations. In this application, the acquisition channel of the training data is not limited. It can be retrieved from local storage, loaded from a remote model library, or generated by adapting existing models.

[0020] It is understandable that the training data is standardized and preprocessed to be uniformly converted into a format that the model can recognize, so as to ensure the normal execution of subsequent word segmentation and hierarchical transmission processes.

[0021] Step 102: Input the data to be trained into the hybrid expert model and obtain the expert routing information of each hybrid expert layer in at least one hybrid expert layer of the hybrid expert model.

[0022] In some embodiments, expert routing information comprises all statistical information related to task allocation within each hybrid expert layer during the training of the hybrid expert model. Core components include the number of tokens (terms) received and processed by the expert network, the token routing allocation probability output by the gating network (term allocation probability), and the number / proportion of expert network activations. This information forms the basis for analyzing routing scheduling. A term is a basic data processing unit generated by the hybrid expert model's processor after processing the input task data into a terminology format. The terms in each hybrid expert layer are those basic terms that, after being routed by the router of the current hybrid expert model, are directed to the hybrid expert layer and processed by the expert networks within that layer.

[0023] In some embodiments, this application does not limit the specific method of obtaining the hybrid expert model, as long as the hybrid expert model that needs to be optimized for load can be obtained. Optionally, obtaining the hybrid expert model in this application includes, but is not limited to: obtaining a pre-stored hybrid expert model from a local storage medium; and / or, loading the hybrid expert model from a remote server or model library; and / or, constructing a hybrid expert model based on an existing pre-trained model; and / or, designing and training a hybrid expert model from scratch.

[0024] Step 103: Based on expert routing information, analyze the routing scheduling of at least one expert network in each hybrid expert layer to determine the routing metrics of each hybrid expert layer.

[0025] In this application, the routing scheduling situation includes at least one of the following: the load balancing situation of at least one expert network, the effective expert ratio, and the routing change situation.

[0026] Each hybrid expert layer is any processing layer in the hybrid expert model that contains a parallel expert network. Each layer independently performs load analysis and structural adjustment to adapt to load differences between layers.

[0027] In some embodiments, this application can collect and analyze the expert routing information of each hybrid expert layer to transform the abstract load distribution into quantifiable and determinable routing indicators, providing an objective basis for subsequent adaptive determination; at the same time, it follows the principle of independent analysis at a single level, that is, each hybrid expert layer collects information, analyzes the load, and generates indicators separately, adapting to the differences in load characteristics of different hybrid expert layers and avoiding one-size-fits-all analysis bias.

[0028] The hybrid expert model comprises a multi-layer neural network structure, with at least one layer being a hybrid expert layer. Each hybrid expert layer includes multiple expert networks and at least one gating network, used to generate the routing probability distribution of the corresponding expert network based on the features of the input sample or token.

[0029] Expert networks are independent sub-networks deployed in parallel within a hybrid expert layer. They are the core units that perform specific computational tasks. Each expert network can independently receive samples / terms assigned by the gating network and complete forward computation. Multiple expert networks have different functions and exist in parallel.

[0030] Routing metrics are quantitative indicators that reflect the balance of resource utilization among different expert layers, calculated based on expert routing information analysis. This application includes at least an expert load imbalance index, an effective expert ratio index, and a routing change rate index, which serve as the basis for determining structural adjustments.

[0031] It should be noted that, in order to achieve the model training optimization of this application, the following parameters need to be set before the model training begins: the size of the expert routing statistics time window (i.e., the preset time window); the thresholds corresponding to various routing dynamic indicators (i.e., the preset equilibrium threshold, the preset utilization threshold, and the preset change threshold); the length of the training warm-up phase Twarm (i.e., the length of the warm-up training phase); the length of the cooling time Tcool after expert reordering or pruning (preset cooling time); and hardware resource constraints (i.e., the minimum number of experts to retain at the device level threshold). Subsequently, pre-training is performed on the hybrid expert model.

[0032] Step 104: Perform structural adjustment on the target hybrid expert layer that meets the structural adjustment conditions for the routing indicators to obtain at least one updated hybrid expert layer, and train the hybrid expert model based on at least one updated hybrid expert layer to obtain the trained hybrid expert model.

[0033] In some embodiments, this application may determine the target hybrid expert layer that meets the structural adjustment conditions among each hybrid expert layer based on routing indicators; and perform structural adjustment on the target hybrid expert layer to obtain at least one updated hybrid expert layer.

[0034] The structural adjustment trigger condition is a preset quantitative rule used to determine whether the hybrid expert layer needs to perform structural adjustment. It is formulated based on the comparison results of routing indicators and corresponding preset thresholds, and adjustment is only triggered for target hybrid expert layers that meet the condition.

[0035] This application can adaptively determine whether each hybrid expert layer needs to perform structural adjustments by comparing routing indicators with preset thresholds, replacing the blind triggering method of fixed period / static rules in related technologies; the determination logic follows the principle of joint determination of multiple indicators to avoid noise interference from a single indicator and improve the accuracy of the determination.

[0036] Specifically, in this application, each type of routing metric can be pre-configured with a corresponding preset threshold (e.g., the imbalance metric corresponds to a preset balance threshold, the effective utilization rate metric corresponds to a preset utilization rate threshold, and the stability change metric corresponds to a preset change threshold). The thresholds can be flexibly set according to the training requirements and model size of the hybrid expert model. That is, each type of routing metric in the hybrid expert layer is compared with its corresponding preset threshold to obtain the judgment result for each type of metric (e.g., whether the expert load imbalance metric exceeds the threshold, whether the effective expert ratio metric is lower than the threshold, and whether the routing change rate metric exceeds the threshold). Based on the preset joint judgment rules, the target hybrid expert layer that meets the structural adjustment triggering conditions is determined.

[0037] In some embodiments, this application can perform targeted structural adjustment operations on target hybrid expert layers that meet the structural adjustment trigger conditions, optimize the task load allocation of expert networks, and improve the resource utilization efficiency and training stability of hybrid expert models. This also follows the principle of independent adjustment at the single layer level, adjusting only the target hybrid expert layer that meets the trigger conditions without affecting the normal training of other hybrid expert layers. Furthermore, the adjustment operation is based on the importance assessment of expert networks to avoid model performance degradation caused by blind adjustments.

[0038] Structural adjustment operations are optimization operations performed on expert networks in a target hybrid expert layer. They include at least one of expert reordering and expert pruning, and can be performed individually or in combination. The aim is to optimize model load and improve resource utilization efficiency. Expert reordering adjusts the order of expert networks on computing devices based on their importance assessment results, prioritizing high-load or high-contribution expert networks on devices with lower communication costs, higher computing power, or more efficient local access. Expert pruning prunes long-term low-load, low-contribution, or redundant expert networks, reducing their number to decrease model parameter size and computational overhead. When determining the final pruning set, hardware resource constraints are introduced: pruning or reordering an expert network will cause the number of expert networks on its device to fall below a preset threshold. If necessary, the pruning or rearrangement decision for the expert network is cancelled or adjusted. System constraints can be further extended to dimensions such as communication bandwidth, memory usage, energy consumption budget, and latency budget to avoid idle local devices or exacerbating system bottlenecks. The aforementioned rearrangement and pruning operations are preferably executed in conjunction with the LAEP (Layer Adaptive Expert Management) strategy; that is, different hybrid expert layers can adopt different expert management strategies, and the pruning ratio, trigger strength, and execution method can all be adjusted independently at the layer level.

[0039] The importance assessment results are quantitative assessments of the load contribution, activation frequency, and computational value of each expert network in the target hybrid expert layer based on expert routing information. These results are used to determine expert networks with long-term low load, low contribution, or redundancy based on the quantitative assessment structure, and serve as a direct basis for performing structural adjustment operations.

[0040] In the embodiments of this application, after performing structural adjustments on the target hybrid expert layer, the hybrid expert model can be directly trained based on the expert network in the updated hybrid expert layer until the model reaches the preset training objective, ensuring training effectiveness and performance stability. The preset training objective may include, but is not limited to, at least one of the following: accuracy or precision reaching the target, loss function convergence, inference latency meeting requirements, throughput reaching a threshold, load balancing meeting expectations, and resource utilization reaching the target. In the embodiments of this application, this objective can be set according to actual circumstances, and no specific limitation is made.

[0041] In summary, this application addresses the problem of blind adjustments triggered by fixed-period, single-indicator adjustments by independently collecting expert routing information from each hybrid expert layer during hybrid expert model training, analyzing routing scheduling, and generating routing indicators reflecting resource utilization balance. It leverages multi-dimensional routing indicators to achieve adaptive and precise determination of structural adjustments, and then performs targeted expert reordering and / or pruning operations based on expert importance assessment. This avoids the waste of computation and communication overhead caused by ineffective adjustments when routing distribution is stable, and can promptly intervene in scenarios of expert load imbalance and unstable routing structure. It achieves precise optimization of inter-layer load in the hybrid expert model, significantly improving the model's resource utilization balance and training stability. Simultaneously, it reduces parameter redundancy caused by redundant experts, lowers cross-device communication costs in distributed training scenarios, and balances model performance with system operating efficiency.

[0042] Figure 2 A flowchart illustrating the second model training optimization method proposed in this application is further shown. Based on Figure 2 The illustrated embodiment further explains step 102. Figure 2 This may include the following steps.

[0043] Step 201: Calculate the number of lexical units and the probability of lexical unit allocation in at least one expert network in each hybrid expert layer within a preset time window.

[0044] Step 202: Integrate the number of lexical units and the probability of lexical unit assignment into the original expert routing information.

[0045] In some embodiments, during model training, within a preset statistical time window, information such as the number of terms selected by the routing network for each expert in each hybrid expert layer, the expert selection probability distribution (i.e., term allocation probability) output by the gating network, the number or proportion of experts being activated, and the changes in expert routing distribution between different statistical windows (such as adjacent statistical windows) are collected, and the above information is integrated to form the original expert routing information for each expert.

[0046] In this application, the expert routing information mainly includes the number of terms and the probability of term assignment. That is, this application counts the number of at least one term and the probability of at least one term assignment of at least one expert network in each hybrid expert layer within a preset time window; and integrates the number of at least one term and the probability of at least one term assignment into at least one original expert routing information.

[0047] Step 203: Perform robustness processing on the original expert routing information to obtain robust expert routing information.

[0048] In some embodiments, since the routing access frequency noise in the original expert routing information is relatively large, directly making model structure adjustment decisions based on this original expert routing information may introduce significant errors. Therefore, after obtaining the original expert routing information of each expert, this application first performs robustness processing on each original expert routing information to obtain at least one expert routing information, thereby obtaining stable routing statistics and reducing the interference of local anomalies or unstable factors on global structure adjustment decisions.

[0049] Specifically, this application robustly corrects at least one original expert routing information, and aggregates the at least one corrected expert routing information across replicas to obtain at least one routing statistic; then, it smooths the at least one routing statistic and performs intra-layer normalization on the at least one smoothed routing statistic, finally obtaining at least one expert routing information.

[0050] The routing access frequency of each expert network can be determined based on at least one original expert routing information, a preset statistical window, and a preset frequency algorithm.

[0051] Optionally, the statistical window in this application can be set to several training iteration steps, several mini-batch data sets, or several parameter update periods, and can be updated using a sliding window method to achieve a balance between statistical accuracy and computational cost. For the e-th expert in layer l, within the t-th statistical period, the algorithm for defining its route access frequency corresponding to the preset frequency is defined as follows: ; in, The original route access frequency of the e-th expert at layer l within the t-th statistical period. This represents the routing function of layer l. This indicates the number of tokens in the current statistics window. For indicator functions, This represents the i-th word within the current statistical window. While the raw statistics of the original expert routing information can be obtained through route access frequency analysis, these statistics are noisy during large-scale training and are not suitable for direct use in structural adjustment decisions.

[0052] In this embodiment, in large-scale distributed training scenarios, routing statistics on different data parallel replicas may differ due to data distribution deviations or transient anomalies. Therefore, this application robustly corrects at least one original expert routing information. In other words, this application can first robustly correct the original expert routing information within each data parallel replica, including but not limited to truncating, scaling, or reducing the weight of abnormally large statistical values ​​to reduce the impact of local anomalies on global decision-making. Specifically, the routing statistics for each data parallel replica are truncated, calculated as follows: ; in, This represents the truncated and corrected expert routing information within the p-th data parallel replica, where p is the index of the data parallel replica. The original route access frequency calculated within the p-th replica. and This represents the upper and lower thresholds for each layer. Its specific value depends on the specific circumstances during training; it can be based on the proportion range of historical exponential moving averages (EMA) or on the theoretical upper limit of the batch size.

[0053] Subsequently, the corrected statistical results are globally summarized through All-Reduce or equivalent aggregation operations on the data parallel dimension to obtain routing statistics that better reflect the actual utilization by global experts: ; in, This represents the global routing statistics after cross-replica aggregation, where P is the number of parallel replicas. Through the above aggregation and robustness processing, routing signal distortion caused by data distribution deviations or transient anomalies in individual replicas can be effectively suppressed.

[0054] To reduce the random fluctuations in route access counts over time, the robustened global routing statistics are time-smoothed. An EMA mechanism is preferred to maintain expert state vectors layer by layer, characterizing the stable utilization trend of each expert over a longer training timescale, rather than just reflecting short-term fluctuations. The calculation method is as follows: ; in, The route statistics signal is smoothed. This is the smoothing coefficient. EMA is maintained independently at each layer, with each layer maintaining a set of EMA state vectors. It can be consistent across all layers, or it can be different for each layer.

[0055] Through the aforementioned time smoothing process, the routing signal of each expert can reflect its stable utilization trend over a longer training timescale, rather than short-term fluctuations. After calculation, smoothed routing statistics are obtained. This application further performs intra-layer normalization to eliminate the impact of batch size changes, retaining only the relative importance information between experts, as follows: ;in, This represents the expert routing information for the e-th expert in layer l after intra-layer normalization. This is the smoothed routing statistics after time smoothing.

[0056] In summary, this application collects expert term carrying capacity and allocation probability within a set statistical window, integrates them to obtain raw routing data, and then performs anomaly correction, multi-copy aggregation, time-series smoothing and intra-layer normalization to remove noise and bias interference, outputting stable and standardized routing information, providing reliable data support for the subsequent determination of routing indicators.

[0057] Figure 3 A flowchart illustrating the third model training optimization method proposed in this application is further shown. Based on Figure 3 The illustrated embodiment further explains step 103. Figure 3 This may include the following steps.

[0058] Step 301: Based on expert routing information, analyze the load balancing status of at least one expert network to determine the load imbalance index; and / or, based on expert routing information, analyze the effective expert ratio of at least one expert network to determine the effective expert ratio index; and / or, based on expert routing information, analyze the routing changes of at least one expert network to determine the routing change rate index.

[0059] In some embodiments, this application sets a preheating training phase length, which represents the length of a buffer phase in the early stages of model training, primarily used for gating network convergence and route allocation stabilization. During this preheating training phase (Twarm), structural adjustment triggering is not performed to avoid false triggering due to the gating network not yet converging or unstable route distribution. Only after the training phase exceeds this preheating training phase is the routing scheduling of each hybrid expert layer analyzed based on expert routing information to obtain routing metrics. That is, this application can determine whether the hybrid expert model is in the preheating training phase during model training based on the preheating training phase length; if the training phase is not in the preheating training phase, the routing scheduling of at least one expert network in each hybrid expert layer is analyzed based on the expert routing information of each hybrid expert layer to obtain routing metrics.

[0060] In some embodiments, this application may employ a multi-signal joint triggering mechanism to avoid excessive noise from a single metric. Therefore, based on the expert routing information after the robustness processing in step 201, the load balancing, effective expert ratio, and routing changes of at least one expert network in each hybrid expert layer can be analyzed to determine the dynamic routing metrics corresponding to each situation.

[0061] In some embodiments, the routing metric includes an expert load imbalance metric. This application can analyze the load balancing of at least one expert network based on expert routing information to determine the load imbalance metric. Specifically, based on the number of at least one term in the expert routing information, the standard deviation and mean of the expert routes for at least one expert network are determined; the routing ratio between the standard deviation and the mean of the expert routes is determined, and this routing ratio is used as the expert load imbalance metric reflecting the degree of load difference among the at least one expert network.

[0062] Optionally, this application can use the coefficient of variation (CV) as an indicator of expert load imbalance to measure the dispersion of load distribution among different expert networks. The coefficient of variation (CV) is defined as: .

[0063] In practical calculations, this application can count the cumulative number of tokens (i.e., the number of terms) for each expert network within a preset time window, and then calculate the CV value. This represents the number of tokens allocated to the e-th expert. Indicates standard deviation, This represents the mean. When all experts have a similar workload, the CV value is close to 0; when a few experts handle most of the tokens and the utilization of the remaining experts is very low, the CV value increases significantly. Furthermore, CV is insensitive to batch size and expert size, making it applicable to models of different sizes.

[0064] When CV exceeds the threshold When the initial triggering condition is met, that is When the initial triggering condition is met, it is determined.

[0065] In some embodiments, the routing metric includes an effective expert ratio metric. This application can analyze the effective expert ratio of at least one expert network based on expert routing information to determine the effective expert ratio metric. Specifically, based on the number of at least one term in the expert routing information and a preset term threshold, the effective number of effective expert networks in at least one expert network is determined; the ratio between the effective number and the total number of expert networks in at least one expert network is determined, and this ratio is used as the effective expert ratio metric reflecting the effectiveness of at least one expert network.

[0066] Optional, the effective expert ratio indicator is used to represent the proportion of experts effectively activated within the statistical window to the total number of experts. The specific formula is: .

[0067] In practical calculations, this application can count the cumulative number of tokens for each expert within a preset time window. The number of tokens received by expert e is... Given a minimum token threshold , E represents the proportion of effective experts, and E represents the total number of experts in the expert network. This means that when most experts are actually used... It will approach 1; when a large number of experts are in a weakly activated state, It will be much less than 1. This indicator can directly reflect the effective utilization rate of experts, consistent with the pruning decision in LAEP.

[0068] when Below the threshold When the initial triggering condition is met, that is When the initial triggering condition is met, it is determined.

[0069] In some embodiments, the routing metric includes a routing change rate metric. This application can analyze the routing changes of at least one expert network based on expert routing information to determine the routing change rate metric. Specifically, based on at least one term allocation probability in the expert routing information, at least one average probability of at least one expert network between adjacent time windows is determined; based on at least one average probability of at least one expert network, a routing change rate metric reflecting the magnitude of routing changes of at least one expert network is determined.

[0070] Optionally, the route change rate metric is a trigger signal based on the route change rate, used to measure the magnitude of change in the expert route distribution between adjacent statistical windows.

[0071] Let the average routing probability (average probability) of expert e within the t-th statistical window be... The rate of change is defined as: .

[0072] This metric calculates the average distribution of expert routes within two adjacent statistical time windows. It can be used to capture sudden changes during the training phase, such as a sudden collapse of the expert route structure that is not detected in time by the load metric. When it is small, it indicates that the routing structure is stable; when... When the routing changes drastically, the distribution of expert tokens becomes unstable.

[0073] when Above the threshold When the initial triggering condition is met, that is When the initial triggering condition is met, it is determined.

[0074] The above metrics can be calculated separately for different hybrid expert layers to reflect the expert routing characteristics at different levels. The expert load imbalance metric and the effective expert ratio metric can reflect expert imbalance and idleness problems in the middle of training, while the routing change rate metric can be used to measure sudden instability.

[0075] In summary, this application collects expert routing information such as the number of terms and the probability of term allocation in each hybrid expert layer during model training using a preset statistical time window. Combined with the warm-up training phase to avoid false triggering when the gating network has not converged, and using a multi-index joint calculation method, it obtains an expert load imbalance index to measure the degree of load dispersion, an effective expert ratio index to reflect the degree of effective expert activation, and a routing change rate index to capture sudden changes in routing distribution. This can comprehensively and accurately characterize the expert routing scheduling of each layer from three dimensions: steady-state imbalance, effective utilization, and dynamic mutation. It effectively reduces the noise interference of a single index, provides a stable, reliable, and hierarchical quantitative judgment basis for subsequent structural adjustments, and improves the accuracy and robustness of adaptive triggering.

[0076] Figure 4 A flowchart illustrating the fourth model training optimization method proposed in this application is further shown. Based on Figure 4 The illustrated embodiment further explains step 104. Figure 4 This may include the following steps.

[0077] Step 401: Compare the expert load imbalance index, effective expert ratio index, and routing change rate index of each hybrid expert layer with the preset balance threshold, preset utilization threshold, and preset change threshold, respectively.

[0078] Step 402: The mixed expert layer that meets any two of the following criteria within a continuous preset time window: expert load imbalance index is greater than preset balance threshold, effective expert ratio index is less than preset utilization threshold, and routing change rate index is greater than preset change threshold is determined as the target mixed expert layer.

[0079] In some embodiments, after calculating the three types of dynamic routing indicators—expert load imbalance, effective expert ratio, and route change rate—each indicator is compared with its corresponding preset threshold, and an adaptive trigger determination is performed. The trigger determination can be set with different thresholds based on hierarchical differences and can be used in conjunction with warm-up phase constraints, protection period constraints, and cooling time constraints.

[0080] The hybrid expert layer is determined to enter the expert reshuffling early warning state when any of the following conditions are met, that is, the hybrid expert layer is determined to be an early warning hybrid expert layer that meets the early warning triggering conditions: The expert load imbalance index is greater than the preset balancing threshold. The proportion of effective experts is less than the preset utilization threshold. The route change rate index is greater than the preset change threshold.

[0081] When a warning state is determined, a warning notification can be generated based on the above indicators for the hybrid expert layer and pushed to the target users. Target users refer to relevant personnel responsible for or involved in hybrid expert model training, model optimization, system operation and maintenance, and algorithm debugging. They are used to receive timely warning information such as expert load imbalance, low expert utilization, or abnormal routing distribution during model training, so as to monitor the model's operating status in real time and take corresponding intervention and debugging measures.

[0082] Furthermore, if at least two of the above three types of indicators simultaneously meet the corresponding early warning conditions within multiple consecutive statistical windows (the number of consecutive preset time windows can be set according to the actual situation and is not limited here), then it is determined that the hybrid expert layer meets the structural adjustment trigger condition, that is, the target hybrid expert layer is obtained, and expert rearrangement and pruning operations can be performed on the target hybrid expert layer.

[0083] Furthermore, when performing expert reordering operations, this application may also employ a tiered response mechanism, selecting intervention methods of varying intensities based on the severity of expert routing anomalies, rather than performing a fixed-intensity reordering operation, thereby avoiding over-adjustment that could interfere with model training. The specific tiers are as follows: Mild response: When the metric only slightly exceeds the threshold and the routing status is not abnormal, only an early warning notification is output and subsequent metric changes are continuously monitored. No actual expert reordering or pruning operations are performed. Medium response: When the metric is obviously abnormal, the expert load is unbalanced, or the effective utilization rate is low, targeted expert reorganization and local adjustments are performed to optimize the load distribution while minimizing the impact on the model structure.

[0084] By employing the aforementioned hierarchical rearrangement strategy, excessive interference with the training process can be significantly reduced while ensuring optimization results.

[0085] In summary, this application compares three types of dynamic routing indicators with corresponding preset thresholds in a hierarchical manner and adopts a joint judgment mechanism that combines single-indicator early warning and multi-indicator continuous satisfaction triggering. This not only enables real-time monitoring and early warning of abnormal states but also effectively avoids false triggering caused by single indicators or instantaneous fluctuations. It significantly improves the accuracy and stability of structural adjustment judgment and can accurately and robustly trigger adaptive adjustments when there is an imbalance in expert load, insufficient effective utilization, or drastic changes in routing distribution. This provides a reliable decision-making basis for subsequent expert reordering and pruning.

[0086] Figure 5 A flowchart illustrating the fifth model training optimization method proposed in this application is further shown. Based on Figure 5 The illustrated embodiment will further explain the process after step 104. Figure 5 This may include the following steps.

[0087] Step 501: Based on the preset cooling time, determine whether the target hybrid expert layer is in a cooling state.

[0088] Step 502: If it is determined that the target hybrid expert layer is in a cooling state, pause the structural adjustment trigger analysis of the target hybrid expert layer.

[0089] In some embodiments, to avoid model training oscillations caused by repeatedly performing expert network structure adjustment operations in a short period of time, this application sets a preset cooling time Tcool (which can be preset in the early stage of model training) after performing expert rearrangement and pruning operations, and enters the corresponding cooling period.

[0090] Specifically, after determining the execution completion time of the structure adjustment operation by at least one expert network in the target mixture-of-experts layer, based on this execution completion time and the preset cooling time Tcool, the cooling state threshold is calculated (i.e., the sum of the execution completion time and Tcool); if the current target time is less than this cooling state threshold, it is determined that the target mixture-of-experts layer is in the cooling state, and in this state, the triggering determination of expert rearrangement and pruning of the target mixture-of-experts layer is suspended, and no structure adjustment operation is triggered; if the current target time is greater than or equal to this cooling state threshold, it is determined that the target mixture-of-experts layer is not in the cooling state, and at this time, the normal training process of the mixture-of-experts model is resumed, and the statistics of expert routing information and the triggering determination of structure adjustment are restarted. Among them, the target time refers to the training time or training step corresponding to the current execution of the structure adjustment triggering determination operation during the model training process, and is used to determine whether the current moment is within the cooling state after the last structure adjustment.

[0091] Briefly speaking, after an expert rearrangement and pruning operation is completed, if the current time t satisfies t < tlast + Tcool (where tlast is the execution completion time of the last structure adjustment), no new rearrangement or pruning operation is triggered, so as to avoid the impact on the stability of model training caused by repeated adjustments in a short period of time.

[0092] In summary, by introducing a cooling state judgment mechanism after the expert network structure adjustment is completed, this application restricts the repeated triggering of structure adjustments within a short period of time according to the preset cooling time, effectively avoiding model training oscillations caused by frequent execution of expert rearrangement and pruning, ensuring that the gating network and the expert network can converge stably, and significantly improving the smoothness and robustness of the model training process.

[0093] Furthermore, to achieve the model load optimization of this application, as Figure 6 shown, this application provides a schematic diagram of a specific mixture-of-experts model training system. The system includes a model training initialization module, a routing statistics collection module, an in-replica robust correction module, a global aggregation module, a time smoothing and normalization module, a metric calculation module, a triggering determination module, a candidate solution generation module, a system constraint correction module, a rearrangement and pruning execution module, and a training phase control module.

[0094] The model training initialization module is used to construct a hybrid expert model and set parameters such as the statistical time window, thresholds for various routing dynamic indicators, the length of the training warm-up phase, the cooling-off time after expert reordering or pruning, and the minimum expert retention threshold at the device level, and then initiates the warm-up phase. The routing statistics acquisition module is used to collect expert routing information within the statistical time window, including the number of terms selected by each expert by the routing network, the expert selection probability distribution (i.e., term allocation probability) output by the gating network, the number of times or proportions of experts being activated, and the changes in expert routing distribution between different statistical windows (such as adjacent statistical windows). The robust correction module within each replica is used to truncate and correct the statistical values ​​within each data parallel replica to suppress local anomalies. The global aggregation module aggregates the correction results of each replica through All-Reduce to obtain global routing statistics. The time smoothing and normalization module uses EMA to smooth the expert routing information over time and then normalizes it within each layer to eliminate short-term fluctuations and batch size effects. The indicator calculation module calculates dynamic indicators such as expert load imbalance, effective expert ratio, and routing change rate. The trigger determination module compares metrics with thresholds to determine whether to trigger a rearrangement or pruning operation. The candidate solution generation module generates adaptive rearrangement and pruning candidate solutions based on the expert network importance evaluation results. The system constraint correction module introduces a minimum number of experts to be retained on the device, adjusting candidate solutions to avoid local bottlenecks. The rearrangement and pruning execution module actually performs expert rearrangement and pruning operations. The training phase control module manages warm-up and cool-down times to avoid early or frequent structural adjustments during training.

[0095] Furthermore, such as Figure 7 As shown, this application provides a schematic diagram of a specific optimization method for model training.

[0096] Reference Figure 7 The optimization process for model training provided in this application starts with building and training a hybrid expert model. It collects the original expert routing information of each hybrid expert layer in the model during training, and performs robust correction within replicas, global aggregation across replicas, temporal smoothing, and intra-layer normalization on this information to obtain expert routing information. Based on this information, dynamic routing metrics reflecting load balancing, effective utilization, and stability are calculated. Adaptive triggering is then performed. If the triggering conditions are not met, training continues and the statistical window is updated. If the triggering conditions are met, system-aware constraint correction is performed, and layer-adaptive expert rearrangement and / or pruning operations are performed on the target hybrid expert layer. Finally, a cooling phase begins, and model training resumes after the cooling phase, returning the collected expert routing information. This achieves a closed-loop load optimization system that optimizes on demand and adjusts precisely.

[0097] Understandably, for Figure 7 For the specific implementation process of each step, please refer to... Figures 1 to 5 The embodiments shown will not be described in detail here.

[0098] To facilitate understanding, the implementation process will be explained in detail below using the specific application scenario of language model pre-training based on a hybrid expert model structure: Set the first 5% to 15% of the training steps (or a fixed number of thousands to tens of thousands of steps) of the model pre-training as a warmup phase to avoid false triggering caused by the randomness of routing in the early stage of training.

[0099] Set the expert routing statistics time window size to 100 to 5000 training steps. When the model parameter size is larger and the number of parallel data paths is greater, it is preferable to use a larger statistics window.

[0100] Set the coefficient of variation threshold =0.5. When the coefficient of variation corresponding to the expert load is greater than 0.5, it is determined that the degree of imbalance of expert load exceeds the preset threshold.

[0101] The minimum number of activation tokens is set to 1% of the average load of experts, i.e., ε = 0.01. Set an effective expert ratio threshold. =0.5, when the percentage of valid experts in the statistical window is less than 50%, it is determined that there is an expert idleness phenomenon.

[0102] Set route change rate threshold =0.2. When the rate of change of the expert average route distribution between adjacent statistical windows is greater than 0.2, it is determined that the route structure has changed significantly.

[0103] The cooldown time after expert rearrangement is set to 2 to 10 times the statistical window to prevent repeated structural adjustments from being performed in a short period of time.

[0104] The robust truncation upper and lower thresholds can be determined based on the historical EMA statistical range, the theoretical upper bound of the batch size, or empirical quantiles. in , .

[0105] The EMA smoothing coefficient λ is preferably set between 0.90 and 0.99. In models with a large number of parameters, a value closer to 1 is preferred to better suppress short-term fluctuations.

[0106] Minimum number of experts to retain at the device level The settings can be configured based on the original number of experts, communication capabilities, and video memory capacity of the equipment to ensure that each device maintains a reasonable load after the trimming. ,in It is the original number of experts on device d. .

[0107] The values ​​mentioned above are only used to enhance the feasibility of this specific example and are not intended to limit the specific values.

[0108] In summary, this application, through an adaptive triggering mechanism and joint judgment of multiple indicators, can accurately grasp the timing of expert reordering, avoiding ineffective adjustments and timely intervention in load imbalance and routing degradation issues. This effectively improves expert utilization, load balance, and overall computing resource utilization efficiency. At the same time, the triggering cooling mechanism reduces the non-stationarity caused by frequent reordering, significantly enhancing the stability of hybrid expert model pre-training. Furthermore, this method is compatible with hierarchical adaptive expert pruning strategies, can be seamlessly integrated into existing training frameworks, and has low engineering implementation cost and strong practicality.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0110] Embodiments of this application also provide an optimization device 800 for model training. Figure 8 This is a schematic diagram of the structure of an optimization device for model training provided in an embodiment of this application, as shown below. Figure 8 As shown, it includes: Acquisition unit 810 is used to acquire the data to be trained; The statistical unit 820 is used to input the data to be trained into the hybrid expert model and to obtain the expert routing information of each hybrid expert layer in at least one hybrid expert layer in the hybrid expert model. Analysis unit 830 is used to analyze the routing scheduling of at least one expert network in each hybrid expert layer based on expert routing information, so as to determine the routing indicators of each hybrid expert layer. The routing scheduling includes at least one of the following: the load balancing status of at least one expert network, the effective expert ratio, and the routing change status. The adjustment unit 840 is used to perform structural adjustment on the target hybrid expert layer that meets the structural adjustment conditions for the routing indicators, to obtain at least one updated hybrid expert layer, and to train the hybrid expert model based on the at least one updated hybrid expert layer, to obtain the trained hybrid expert model.

[0111] Furthermore, in one possible implementation of this application embodiment, the statistical unit 820 is used for: Statistically analyze the number of at least one word and the probability of at least one word assignment in at least one expert network within a preset time window in each hybrid expert layer; The number of at least one lexical unit and the probability of at least one lexical unit assignment are integrated into the original expert routing information; The original expert routing information is robustened to obtain robust expert routing information.

[0112] Furthermore, in one possible implementation of this application embodiment, the statistics unit 820 is used to: robustly correct the original expert routing information, and aggregate the robustly corrected expert routing information across replicas to obtain routing statistics information; smooth the routing statistics information, and perform intra-layer normalization processing on the smoothed routing statistics information to obtain expert routing information.

[0113] Furthermore, in one possible implementation of this application embodiment, the analysis unit 830 is used to: determine whether the training phase of the hybrid expert model is in the preheating training phase based on the length of the preheating training phase; if the training phase is not in the preheating training phase, analyze the routing scheduling of at least one expert network in each hybrid expert layer based on at least one expert routing information, so as to obtain the routing index of each hybrid expert layer.

[0114] Furthermore, in one possible implementation of this application embodiment, the analysis unit 830 is configured to: analyze the load balancing status of at least one expert network based on expert routing information to determine a load imbalance index; and / or, analyze the effective expert ratio of at least one expert network based on expert routing information to determine an effective expert ratio index; and / or, analyze the routing changes of at least one expert network based on expert routing information to determine a routing change rate index.

[0115] Furthermore, in one possible implementation of this application embodiment, the analysis unit 830 is configured to: determine the expert routing standard deviation and expert routing mean of at least one expert network based on the number of at least one term in the expert routing information; determine the routing ratio between the expert routing standard deviation and the expert routing mean, and determine the routing ratio as an expert load imbalance index reflecting the degree of load difference of at least one expert network.

[0116] Furthermore, in one possible implementation of this application embodiment, the analysis unit 830 is configured to: determine the effective number of effective expert networks in at least one expert network based on the number of at least one term in the expert routing information and a preset term threshold; determine the ratio between the effective number and the total number of expert networks in at least one expert network, and determine the ratio as an effective expert ratio index reflecting the effectiveness of at least one expert network.

[0117] Furthermore, in one possible implementation of this application embodiment, the analysis unit 830 is configured to: determine at least one average probability of at least one expert network between adjacent time windows based on at least one token allocation probability in the expert routing information; and determine a routing change rate index reflecting the routing change magnitude of at least one expert network based on at least one average probability of at least one expert network.

[0118] Furthermore, in one possible implementation of the embodiments of this application, the adjustment unit 840 is used to: determine the target hybrid expert layer that meets the structural adjustment conditions among each hybrid expert layer based on the routing index; and perform structural adjustment on the target hybrid expert layer to obtain at least one updated hybrid expert layer.

[0119] Furthermore, in one possible implementation of this application embodiment, the routing metrics include an expert load imbalance metric, an effective expert ratio metric, and a routing change rate metric. The adjustment unit 840 is used to: compare the expert load imbalance metric, the effective expert ratio metric, and the routing change rate metric of each hybrid expert layer with a preset balancing threshold, a preset utilization threshold, and a preset change threshold, respectively; and determine the hybrid expert layer that satisfies any two of the following within a continuous preset time window: the expert load imbalance metric is greater than the preset balancing threshold, the effective expert ratio metric is less than the preset utilization threshold, and the routing change rate metric is greater than the preset change threshold, as the target hybrid expert layer.

[0120] Further, in one possible implementation of this application embodiment, the adjustment unit 840 is used to: after comparing the expert load imbalance index, the effective expert ratio index, and the routing change rate index with the preset balancing threshold, the preset utilization threshold, and the preset change threshold respectively, determine the mixed expert layer that meets any one of the following conditions as the early warning mixed expert layer that meets the early warning triggering condition; generate early warning notification information based on the expert load imbalance index, the effective expert ratio index, or the routing change rate index of the early warning mixed expert layer, so as to push the early warning notification information to the target user.

[0121] Furthermore, in one possible implementation of this application embodiment, the adjustment unit 840 is configured to: after performing structural adjustment on the target hybrid expert layer whose routing indicators meet the structural adjustment conditions to obtain at least one updated hybrid expert layer, determine whether the target hybrid expert layer is in a cooling state based on a preset cooling time; and if it is determined that the target hybrid expert layer is in a cooling state, suspend the structural adjustment trigger analysis of the target hybrid expert layer.

[0122] Furthermore, in one possible implementation of this application embodiment, the adjustment unit 840 is used to: determine the execution completion time of the structural adjustment of the target hybrid expert layer; determine a cooling state threshold based on the execution completion time and a preset cooling time; determine that the target hybrid expert layer is in a cooling state when the target time is less than the cooling state threshold; and determine that the target hybrid expert layer is not in a cooling state when the target time is greater than or equal to the cooling state threshold.

[0123] Furthermore, in one possible implementation of this application embodiment, the adjustment unit 840 is used to: determine the importance assessment result of at least one expert network of the target hybrid expert layer based on the expert routing information of the target hybrid expert layer; and adjust the structure of the target hybrid expert layer according to the importance assessment result and hardware resource constraints to obtain at least one updated hybrid expert layer.

[0124] Furthermore, in one possible implementation of this application embodiment, the adjustment unit 840 is configured to: determine at least one expert network to be processed in at least one expert network in the target hybrid expert layer based on the importance assessment result; analyze the hardware resource situation corresponding to at least one expert network to be processed based on hardware resource constraints, and obtain the analysis result; based on the analysis result, select target expert networks that meet the hardware resource constraints from at least one expert network to be processed, and rearrange and / or prune the target expert networks to obtain at least one updated hybrid expert layer.

[0125] For a description of the features in the embodiment corresponding to the optimization device for model training, please refer to the relevant description in the embodiment corresponding to the optimization method for model training, which will not be repeated here.

[0126] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described optimization method embodiments for model training.

[0127] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described optimization method embodiments for model training at runtime.

[0128] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0129] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described optimization method embodiments for model training.

[0130] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described optimization method embodiments for model training.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The above provides a detailed description of the optimization method, electronic device, storage medium, and product for model training provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. An optimization method for model training, characterized in that, include: Obtain the training data; The training data is input into the hybrid expert model, and the expert routing information of each hybrid expert layer in at least one hybrid expert layer of the hybrid expert model is obtained; Based on the expert routing information, the routing scheduling of at least one expert network in each hybrid expert layer is analyzed to determine the routing metrics of each hybrid expert layer. The routing scheduling includes at least one of the following: load balancing status, effective expert ratio, and routing change status of the at least one expert network. The routing metrics include at least one of the following: expert load imbalance index, effective expert ratio index, and routing change rate index. The expert load imbalance index is obtained by analyzing the load balancing status, the effective expert ratio index is obtained by analyzing the effective expert ratio, and the routing change rate index is obtained by analyzing the routing change status. The expert load imbalance index, effective expert ratio index, and routing change rate index of each hybrid expert layer are compared with preset balancing thresholds, preset utilization thresholds, and preset change thresholds, respectively. The hybrid expert layer that satisfies any two of the following conditions within a consecutive preset time window: expert load imbalance index greater than the preset balancing threshold, effective expert ratio index less than the preset utilization threshold, and routing change rate index greater than the preset change threshold, is determined as the target hybrid expert layer. The target hybrid expert layer whose routing index meets the structural adjustment conditions is structurally adjusted to obtain at least one updated hybrid expert layer, and the hybrid expert model is trained based on the at least one updated hybrid expert layer to obtain the trained hybrid expert model. Wherein, obtaining the expert routing information of each hybrid expert layer in at least one hybrid expert layer of the hybrid expert model includes: The number of at least one word and the probability of at least one word assignment in at least one expert network in each hybrid expert layer within a preset time window are statistically analyzed. The number of at least one lexical unit and the probability of assignment of at least one lexical unit are integrated into the original expert routing information; The original expert routing information is then subjected to robustness processing to obtain robust expert routing information.

2. The method according to claim 1, characterized in that, The robustening process performed on the original expert routing information to obtain robust expert routing information includes: The original expert routing information is robustly corrected, and the corrected expert routing information is aggregated across replicas to obtain routing statistics. The routing statistics are smoothed, and the smoothed routing statistics are then normalized within a layer to obtain the expert routing information.

3. The method according to claim 1, characterized in that, The step of analyzing the routing scheduling of at least one expert network in each hybrid expert layer based on the expert routing information, in order to determine the routing metrics of each hybrid expert layer, includes: Based on the length of the preheating training phase, determine whether the hybrid expert model is in the preheating training phase during the model training process; If the training phase is not in the preheating training phase, the routing scheduling of at least one expert network in each hybrid expert layer is analyzed based on the expert routing information to obtain the routing index of each hybrid expert layer.

4. The method according to claim 1, characterized in that, The step of analyzing the routing scheduling of at least one expert network in each hybrid expert layer based on the expert routing information, in order to determine the routing metrics of each hybrid expert layer, includes: Based on the expert routing information, the load balancing status of the at least one expert network is analyzed to determine the load imbalance index. Based on the expert routing information, the effective expert ratio of the at least one expert network is analyzed to determine the effective expert ratio index. Based on the expert routing information, the routing changes of the at least one expert network are analyzed to determine the routing change rate index.

5. The method according to claim 3, characterized in that, The step of analyzing the load balancing status of the at least one expert network based on the expert routing information to determine the load imbalance index includes: Based on the number of at least one term in the expert routing information, determine the expert routing standard deviation and expert routing mean of the at least one expert network; Determine the routing ratio between the standard deviation of the expert routes and the mean of the expert routes, and use the routing ratio as an expert load imbalance index that reflects the degree of load difference in the at least one expert network.

6. The method according to claim 3, characterized in that, The step of analyzing the effective expert ratio of the at least one expert network based on the expert routing information to determine the effective expert ratio index includes: Based on the number of at least one term in the expert routing information and a preset term threshold, determine the effective number of effective expert networks in the at least one expert network; Determine the ratio between the effective number and the total number of expert networks in the at least one expert network, and define the ratio as the effective expert ratio index reflecting the effectiveness of the at least one expert network.

7. The method according to claim 3, characterized in that, The step of analyzing the routing changes of the at least one expert network based on the expert routing information to determine the routing change rate index includes: Based on at least one term allocation probability in the expert routing information, determine at least one average probability of the at least one expert network between adjacent time windows; Based on at least one average probability of the at least one expert network, a routing change rate index reflecting the magnitude of routing changes in the at least one expert network is determined.

8. The method according to claim 1, characterized in that, The step of structurally adjusting the target hybrid expert layer whose routing metrics meet the structural adjustment conditions to obtain at least one updated hybrid expert layer includes: Based on the routing metrics, the target hybrid expert layer that meets the structural adjustment conditions is determined among the hybrid expert layers; The target hybrid expert layer is structurally adjusted to obtain the at least one updated hybrid expert layer.

9. The method according to claim 8, characterized in that, The method includes comparing the expert load imbalance index, the effective expert ratio index, and the routing change rate index of each hybrid expert layer with preset balance thresholds, preset utilization thresholds, and preset change thresholds, respectively. A hybrid expert layer that meets any one of the following criteria is defined as a pre-warning hybrid expert layer that meets the pre-warning triggering conditions: the expert load imbalance index is greater than the preset balance threshold, the effective expert ratio index is less than the preset utilization threshold, and the routing change rate index is greater than the preset change threshold. Based on the expert load imbalance index, effective expert ratio index, or routing change rate index of the aforementioned early warning hybrid expert layer, an early warning notification is generated and pushed to the target user.

10. The method according to claim 1, characterized in that, After adjusting the structure of the target hybrid expert layer that meets the structural adjustment conditions for the routing indicators to obtain at least one updated hybrid expert layer, the method includes: Based on a preset cooling time, determine whether the target hybrid expert layer is in a cooling state; If it is determined that the target hybrid expert layer is in the cooled state, the structural adjustment trigger analysis of the target hybrid expert layer is paused.

11. The method according to claim 10, characterized in that, The step of determining whether the target hybrid expert layer is in a cooling state based on a preset cooling time includes: Determine the completion time for the structural adjustment of the target hybrid expert layer; A cooling state threshold is determined based on the execution completion time and the preset cooling time; If the target time is less than the cooling state threshold, the target hybrid expert layer is determined to be in the cooling state; If the target time is greater than or equal to the cooling state threshold, it is determined that the target hybrid expert layer is not in the cooling state.

12. The method according to claim 1, characterized in that, The step of structurally adjusting the target hybrid expert layer whose routing metrics meet the structural adjustment conditions to obtain at least one updated hybrid expert layer includes: Based on the expert routing information of the target hybrid expert layer, determine the importance assessment result of at least one expert network of the target hybrid expert layer; Based on the importance assessment results and hardware resource constraints, the target hybrid expert layer is structurally adjusted to obtain at least one updated hybrid expert layer.

13. The method according to claim 12, characterized in that, The step of structurally adjusting the target hybrid expert layer based on the importance assessment results and hardware resource constraints to obtain the at least one updated hybrid expert layer includes: Based on the importance assessment results, at least one expert network to be processed is determined from at least one expert network in the target hybrid expert layer; Based on the hardware resource constraints, the hardware resource situation corresponding to the at least one expert network to be processed is analyzed to obtain the analysis results. Based on the analysis results, target expert networks that satisfy the hardware resource constraints are selected from the at least one expert network to be processed, and the target expert networks are rearranged and / or pruned to obtain the at least one updated hybrid expert layer.

14. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the optimization method for model training according to any one of claims 1-13.

Images