Efficient Fault-Tolerant Method for Large-Scale Model Training with Multi-Agent Collaboration

Abstract

This invention relates to the field of distributed machine learning technology and discloses an efficient and fault-tolerant method for the entire training process of large-scale models through multi-agent collaboration. The method includes: estimating the network latency status of each region; recording sample routing decision information and caching it to local edge coordination nodes; statistically analyzing local expert activation information and calculating expert importance scores within the region; performing latency compensation fusion on the expert importance scores of each region to generate a global expert importance score; performing selective recalculation within the region based on the routing decision cache to generate accurate local compensation gradients; and aggregating gradients from various sources to update global model parameters. This invention solves the technical problems of inaccurate expert contribution evaluation and recalculation latency exceeding the fault tolerance window in cross-regional hybrid expert model training.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning technology, and more specifically, to an efficient fault-tolerant method for the entire training process of large models with multi-agent collaboration. Background Technology

[0002] In multi-agent collaborative large-scale model training scenarios deployed across geographically distributed data centers, hybrid expert architectures are widely adopted to improve model capacity and computational efficiency. Different expert subnetworks are deployed in each geographically distributed data center. Input samples are dynamically routed through a gating network to activate at least a few experts for processing. Each edge coordination node is responsible for the state management of its local region, while the central coordination node is responsible for global aggregation and cross-regional coordination.

[0003] Existing technologies use a central coordination node to collect activation status information of experts in various regions and calculate expert importance scores for fault prediction and recalculation decisions. When an agent fails, it performs selective recalculation by transmitting routing decision information across regions.

[0004] However, existing technologies have the following drawbacks: First, the cross-regional transmission of expert activation status information has a wide area network latency of tens to hundreds of milliseconds, and the latency fluctuations exhibit non-stationary characteristics. This causes the activation statistics used by the central coordination node to calculate the expert importance score to become outdated, failing to accurately reflect the gradient importance differences at the expert level. Second, the latency of cross-regional transmission of routing decision information may exceed the fault tolerance window, resulting in recalculation not being completed in a timely manner. Third, edge coordination nodes only possess activation information of local experts and lack contribution information of experts from other regions, making it impossible to independently complete the global expert contribution assessment, thus affecting the accuracy of fault prediction and recalculation decisions. Summary of the Invention

[0005] This invention provides an efficient and fault-tolerant method for the entire training process of large-scale models with multi-agent collaboration, solving the technical problems of inaccurate evaluation of expert contributions, imprecise gradient compensation, and low training efficiency caused by network latency differences and node failures in the training of cross-regional hybrid expert models.

[0006] This invention discloses an efficient and fault-tolerant method for the entire training process of a large-scale model with multi-agent collaboration, comprising the following steps: A central coordination node measures the network round-trip delay between each regional data center and the central coordination node; a Kalman filter algorithm is used to filter the delay time-series data of each link, estimating the mean delay and variance of each link, and generating a regional-level network delay state estimate; During the local gradient calculation process, agents within each region record the gated routing decisions triggered by each sample, generating routing decision records, and transmitting these records to the local edge coordination node, which stores the received routing decision records in a routing decision cache; Based on the routing decision records of the current batch in the routing decision cache, the edge coordination node calculates the activation frequency and cumulative routing weight of each expert in the current batch, calculates the gradient energy percentage of each expert, and generates an expert importance score vector within the region; the central coordination node... The adjustment point receives time-stamped expert importance score vectors from each edge coordination node. Based on the regional network latency state estimation, it corrects and aligns the regional expert importance score vectors for each region using timestamps. Confidence weights are assigned to each region based on latency fluctuation variance. The corrected regional expert importance score vectors are then weighted and fused to generate a global expert importance score. When an edge coordination node detects a failure in a local agent, it extracts the routing decision record of the failed agent's processing sample from the routing decision cache. A target agent is selected within the local region. The target agent performs deterministic forward propagation and gradient calculation based on the activated expert index set and routing weights in the routing decision record, generating a precise local compensation gradient. The central coordination node aggregates the precise local compensation gradients and remote compensation gradients from each region, performs global gradient aggregation with the gradients reported by normal agents, and updates the model parameters.

[0007] Furthermore, the Kalman filter algorithm employs adaptive process noise covariance, dynamically adjusting the process noise variance based on the statistical characteristics of the observation residuals. The process noise variance is obtained by multiplying the process noise variance of the previous time step by the smoothing coefficient, plus the product of the square of the observation residual at the current time step and the complement of the smoothing coefficient. The observation residual represents the difference between the actual observation delay value and the Kalman filter predicted delay value at the current time step.

[0008] Furthermore, the routing decision record includes a sample index, an activation expert index set, a routing weight corresponding to each activation expert, and a routing decision generation timestamp; the edge coordination node adopts a sliding window management strategy for the routing decision cache, retaining only the routing decision records of the most recent training batches. The number of batches retained is determined by dividing the fault tolerance window duration by the average processing time of a single batch and rounding up.

[0009] Furthermore, the method for calculating the regional expert importance score is as follows: the product of the activation frequency and average routing weight of each expert in the local region is divided by the sum of the products of the activation frequency and average routing weight of all experts in the local region to obtain the regional expert importance score of each expert; wherein, the average routing weight is the sum of the routing weights of all activated instances of each expert in the current batch divided by the activation frequency.

[0010] Furthermore, the calculation of the regional expert importance score also incorporates the gradient norm statistics of each expert. The product of each expert's activation frequency, average routing weight, and gradient norm statistics is divided by the sum of the products of all experts in the local area to obtain the regional expert importance score of each expert. The gradient norm statistics are the average L2 norm of the gradients generated by each expert in the current batch.

[0011] Furthermore, the timestamp correction and alignment process is as follows: for the regional expert importance score vector of each region, the original timestamp is added to the estimated delay mean value of the corresponding region to obtain the corrected equivalent timestamp, and the time deviation of each region is calculated based on the current time; the confidence weight is determined by the product of the inverse of the delay fluctuation variance and the time decay factor, and the time decay factor is an exponential function value with the natural constant as the base and the negative value of the product of the time decay coefficient and the time deviation as the exponent.

[0012] Furthermore, the fusion method for the global expert importance score is as follows: for each expert, the sum of the products of the confidence weights of each region where the expert is deployed and the expert importance score in the corresponding region is divided by the sum of the confidence weights of each region where the expert is deployed to obtain the global expert importance score for that expert.

[0013] Furthermore, the target agent performs deterministic forward propagation and gradient calculation as follows: based on the set of activated expert indexes in the routing decision record, only the specified expert subnetwork is activated for forward propagation calculation, and the outputs of each activated expert are weighted and combined according to the routing weights to obtain the final output. Then, backpropagation is performed to calculate the gradient; wherein, the weighted combination is the sum of the products of the outputs of each activated expert and their corresponding routing weights.

[0014] Furthermore, before performing selective recalculation, the edge coordination node prioritizes the samples processed by the faulty agent based on the global expert importance score. The recalculation priority of each sample is the sum of the products of the global expert importance scores of all experts activated by the sample and the corresponding routing weights. The recalculation tasks are scheduled in descending order of recalculation priority.

[0015] Furthermore, when the remote compensation gradient fails to return within a preset time limit, the central coordination node generates an approximate compensation gradient based on the average gradient statistics of remote regional experts in historical batches and the routing weight information of the current batch; the approximate compensation gradient is obtained by multiplying the historical average gradient by the scaling ratio, where the scaling ratio is the sum of the routing weights of the current batch divided by the sum of the historical average routing weights.

[0016] This invention discloses a high-efficiency fault-tolerant system for the entire training process of a large-scale model with multi-agent collaboration, comprising: a latency state estimation module, used to measure the network round-trip latency between regional data centers and the central coordination node, using a Kalman filter algorithm to filter the latency time-series data of each link, estimating the mean latency and variance of each link, and generating a regional-level network latency state estimate; a routing decision caching module, used to record the gated routing decisions triggered by each sample during the local gradient calculation process of agents within each region, generating routing decision records and storing them in the routing decision cache; a regional scoring calculation module, used to calculate the gradient energy proportion of each expert in the current batch based on the routing decision records of the current batch in the routing decision cache, and generate a regional expert importance score vector; and latency compensation fusion; The fusion module receives time-stamped expert importance score vectors from each edge coordination node, performs timestamp correction and alignment based on the regional network latency state estimate, sets confidence weights based on latency fluctuation variance, and performs weighted fusion on the corrected regional expert importance score vectors to generate a global expert importance score. The route consistency recalculation module is used to extract the route decision records of the failed agent's processing samples from the route decision cache when a failure is detected in the local domain. It selects the target agent and performs deterministic forward propagation and gradient calculation based on the set of activated expert indices and route weights in the route decision records to generate the local domain accurate compensation gradient. The global gradient aggregation module is used to summarize the local domain accurate compensation gradients and remote compensation gradients from each region, perform global gradient aggregation with the gradients reported by normal agents, and update the model parameters.

[0017] This invention obtains the mean delay and variance of each link through regional network delay state estimation. During the global expert importance score fusion process, it performs timestamp correction and alignment on the expert importance score vectors reported by each region, and performs weighted fusion based on confidence weights set according to the delay variance. This solves the technical problem of inaccurate expert contribution assessment due to outdated information, achieving the technical effect of maintaining the accuracy of global expert importance scores even in non-stationary environments with delay fluctuations. Furthermore, by using a routing decision caching mechanism in edge coordination nodes, the routing decision information required for selective recalculation is pre-stored within the region. Through a routing consistency selective recalculation mechanism, the target agent directly uses the cached routing decisions to perform deterministic calculations, solving the technical problem that the delay of cross-regional transmission of routing decision information may exceed the fault tolerance window. This achieves the technical effect of shortening the critical path recalculation from cross-regional communication to intra-regional communication. Finally, by using a hierarchical expert importance score mechanism, edge coordination nodes can obtain expert contribution information from a global view, solving the technical problem that edge coordination nodes cannot independently complete global expert contribution assessment, and achieving the technical effect of improving the accuracy of fault prediction and recalculation decisions. Attached Figure Description

[0018] Figure 1 This is a flowchart of an efficient fault-tolerant method for the entire training process of a large model with multi-agent collaboration, provided in an embodiment of the present invention. Detailed Implementation

[0019] In multi-agent collaborative large-scale model training scenarios deployed across geographically distributed data centers, the Mixture of Experts (MoE) architecture is widely adopted to improve model capacity and computational efficiency. In this architecture, different expert sub-networks are deployed in each geographically distributed data center. Input samples are dynamically routed through a gating network to at least a few activated experts for processing. Each edge coordination node is responsible for local domain state management, while the central coordination node is responsible for global aggregation and cross-regional coordination.

[0020] However, deploying hybrid expert models across regions faces the following technical challenges: First, the cross-regional transmission of expert activation status information has a wide area network delay of tens to hundreds of milliseconds, and the delay fluctuation is affected by factors such as network congestion and routing changes, exhibiting non-stationary characteristics. This causes the activation statistics information used by the central coordination node to calculate the expert importance score to be outdated and unable to accurately reflect the gradient importance differences at the expert level.

[0021] Second, when an agent malfunctions and requires selective recalculation, the original routing decision must be obtained to reproduce the same expert activation path. However, the delay in transmitting routing decision information across regions may exceed the fault tolerance window, causing the recalculation to fail to be completed in a timely manner.

[0022] Third, edge coordination nodes only have activation information of local experts and lack contribution information of experts from other regions. They cannot independently complete the global expert contribution assessment, which affects the accuracy of fault prediction and recalculation decisions.

[0023] According to an embodiment of this implementation, a highly efficient and fault-tolerant method for the entire training process of a large model through multi-agent collaboration is provided. It should be understood that the implementing entities of this implementation include edge coordination nodes deployed in various regional data centers and a central coordination node responsible for global coordination. Each regional data center deploys several agents and expert sub-networks.

[0024] This embodiment provides an efficient and fault-tolerant method for the entire training process of a large model with multi-agent collaboration, which includes the following steps: Step 1: Estimate the network latency status in each region and generate the mean latency and variance of the fluctuation. The central coordination node measures the network round-trip delay between the regional data centers and the central coordination node. It uses the Kalman filter algorithm to filter the delay time series data of each link, estimates the mean delay and fluctuation variance of each link, and generates a regional network delay status estimate.

[0025] It should be noted that the regional network latency state estimation includes the estimated mean latency values ​​for each region. And volatility variance estimate ,in This indicates a geographic index.

[0026] Furthermore, the network round-trip delay is measured as follows: the central coordinating node sends a timestamped probe message to the edge coordinating nodes in each regional data center. After receiving the probe message, the edge coordinating node immediately returns a response message. The central coordinating node records the time difference between the probe message sending time and the response message receiving time as the round-trip delay observation value. The measurement period is set to perform a delay measurement once per training batch.

[0027] Furthermore, to adapt to the non-stationary nature of wide area network delays, the Kalman filter algorithm employs adaptive process noise covariance, dynamically adjusting the process noise variance based on the statistical characteristics of the observed residuals. : in, Indicates the first Location at any given moment The observation residuals Represents the smoothing coefficient, with a value range of , indicating the th . Location at any given moment The process noise variance.

[0028] Furthermore, observe residuals Indicates the first Location at any given moment The difference between the actual observed delay value and the Kalman filter predicted delay value is the observation residual, which reflects the degree of abrupt change in delay. When the observation residual is large, the process noise variance is increased to improve the response speed of the Kalman filter algorithm to delay changes.

[0029] Step 2: Record the routing decision information of the sample and cache it to the local edge coordination node. During the local gradient calculation process, agents in each region record the gated routing decisions triggered by each sample, generate routing decision records, and transmit the routing decision records to the local edge coordination node. The edge coordination node stores the received routing decision records in the routing decision cache.

[0030] It should be noted that the routing decision record includes a sample index, a set of activation expert indexes, the routing weights corresponding to each activation expert, and a timestamp indicating the generation of the routing decision. For the... For each sample, its routing decision record is represented as follows: ,in Indicates the sample index. Indicates that it is activated A set of indexes for experts, where They represent the first The first to the second of the samples An index of activated experts, This represents the corresponding route weight, where They represent the first The first to the second of the samples The routing weight corresponding to each activated expert. Indicates the timestamp of the routing decision generation. This indicates the number of experts activated for each sample.

[0031] Furthermore, to control the storage overhead of the routing decision cache, the edge coordination node adopts a sliding window management strategy for the routing decision cache, retaining only the most recently accessed data. The routing decision records of each training batch, among which The calculation method is determined based on the fault tolerance window duration and batch processing rate. , Indicates the duration of the fault tolerance window. This indicates the average processing time for a single batch.

[0032] Furthermore, the fault tolerance window duration This represents the maximum allowable time interval from the occurrence of an agent failure to the necessary completion of gradient recalculation and aggregation. The fault tolerance window duration is determined by the training system's synchronization protocol and batch update cycle. The batch processing rate represents the number of training batches completed per unit time. The sliding window management strategy retains the number of batches. Ensure coverage of all samples that may need to be recalculated within the fault tolerance window.

[0033] Furthermore, the gated network adopts a multilayer perceptron structure. The input layer receives the feature representation of the samples, the hidden layer uses the ReLU activation function for nonlinear transformation, and the output layer is a fully connected layer with an output dimension equal to the total number of experts. The output shows the distribution of routing weights for each expert, which is then normalized using the Softmax normalization function before selecting the expert with the highest weight. One expert serves as the activation expert.

[0034] Furthermore, the gating network is trained using an end-to-end supervised learning approach, jointly trained with the expert sub-network. The loss function comprises a weighted combination of task loss and load balancing loss. Task loss measures the difference between the model output and the true label, while load balancing loss measures the balance of activation frequencies among experts. These two losses are combined into a total loss function using adjustable weight coefficients, and the Adam optimization algorithm is used for parameter updates.

[0035] Step 3: Collect activation information of local experts and calculate the importance score of experts within the region. The edge coordination node calculates the gradient energy percentage of each expert in the current batch based on the routing decision records in the routing decision cache, and generates an expert importance score vector within the region.

[0036] It should be noted that for locally deployed [number]... Each expert, and their regional expert importance score. The calculation method is as follows: in, Indicates the first The activation frequency of each expert in the current batch. Indicates the first The average routing weight of the expert is calculated as follows: The sum of the routing weights of all active instances in the current batch by the activation frequency is calculated by dividing the sum of the routing weights of each expert by the activation frequency. This represents the set of expert indexes deployed locally. This represents the collection of expert indexes deployed locally. Index variables for each expert in China.

[0037] Furthermore, activation frequency This indicates that among all samples processed in the current batch, the [number]th [sample] is [the [number]th [sample]. The number of samples in which an expert is selected by the gating network to participate in the computation; the activation frequency reflects the frequency of expert usage; and the average routing weight. Indicates the first The average weight value obtained by each expert when they are activated determines the proportion of contribution of the expert's output to the final result.

[0038] Furthermore, to more accurately reflect the experts' contributions to gradient updates, the calculation of gradient energy proportion also incorporates the gradient norm information generated by each expert's processed samples. Specifically, the edge coordination node obtains the gradient norm statistics of each expert from the local agent. The gradient energy ratio is defined as: in, Indicates the first The average L2 norm of the gradients generated by each expert in the current batch. Indicates the first The average L2 norm of the gradients generated by each expert in the current batch.

[0039] Furthermore, the gradient norm statistics are obtained as follows: after the agent performs backpropagation to calculate the gradients of each expert, it calculates the L2 norm of the gradient tensor of each expert, and then calculates the L2 norm of the gradient tensor of the current batch. The average of the L2 norms of the gradients of all activated instances of the expert is used to generate the first... The gradient norm statistics of each expert are reported to the edge coordination node along with the routing decision records.

[0040] Furthermore, the L2 norm of the gradient represents the square root of the sum of squares of all elements in the gradient tensor. The L2 norm of the gradient measures the overall magnitude of the gradient. Experts with larger gradient norms have a more significant impact on model parameter updates, and therefore should receive higher weight in the regional expert importance score.

[0041] Step 4: Perform delayed compensation fusion on the regional expert importance scores of each region to generate a global expert importance score. The central coordination node receives the time-stamped expert importance score vectors reported by each edge coordination node. Based on the regional network latency state estimation, the central coordination node corrects and aligns the regional expert importance score vectors of each region with timestamps. Based on the latency fluctuation variance, the central coordination node sets confidence weights for the regional expert importance scores of each region. The central coordination node then performs weighted fusion on the corrected regional expert importance score vectors to generate a global expert importance score.

[0042] It should be noted that the timestamp correction and alignment is handled as follows: For timestamps from the... The regional expert importance score vector for each region, with its original timestamp being... The corrected equivalent timestamp is ,in For the first The estimated mean delay for each region. The central coordination node uses the current time... Based on this, the time bias of expert importance scores within each region is calculated. .

[0043] Furthermore, the original timestamp This represents the moment when the edge coordination node generates the expert importance score vector within the region, represented by the equivalent timestamp. This represents the expected time when the expert importance score vectors within a region reach the central coordination node, taking into account network transmission delays; the time deviation. This indicates the actual time lag of expert importance rating information within a region; the greater the time lag, the more outdated the expert importance rating information within the region.

[0044] It should be noted that the confidence weights are set as follows: For the first... Each region, its confidence weight With the The variance of delayed fluctuations in different regions is negatively correlated: in, Indicates the first Estimates of the variance of delay fluctuations in each region. To prevent small constants from being divided by zero, the value is taken as... , This is the time decay coefficient, which controls the rate at which the time deviation decays on the confidence weight. Due to time deviation, This represents an exponential function with the natural constant as its base.

[0045] Furthermore, confidence weights It consists of two parts: Part 1 Reflecting the impact of latency stability, the smaller the latency fluctuation variance, the more stable the network transmission. The higher the reliability of the expert importance score within a region, the better; Part Two Reflecting the impact of time freshness, the greater the time deviation, the more outdated the expert importance rating information within the region, and its confidence weight decays exponentially.

[0046] Furthermore, the time decay coefficient The settings are adaptively configured based on the time sensitivity of the training batch, and the calculation method is as follows: ,in Represents the natural logarithm function. This represents the time skew threshold required for the confidence weights to decay to half of their initial values. The time skew threshold is determined based on the convergence speed of the model training and the batch update frequency.

[0047] Furthermore, time sensitivity represents the rate at which the importance score of experts within a region changes as the training process progresses. When the model training is in the rapid convergence phase, the expert activation pattern changes rapidly, and the time sensitivity is high. A larger time decay coefficient needs to be set to reduce the weight of outdated information. When the model training tends to stabilize, the expert activation pattern changes slowly, and the time decay coefficient can be appropriately reduced.

[0048] It should be noted that the fusion method for the global expert importance score is as follows: For the global [number] [number] [point], the fusion method is as follows: Each expert's global expert importance score. The calculation is as follows: in, Indicates that there is a deployment of the first A regional collection of experts This indicates traversing a set of regions. Index variables for various regions in China, Indicates the first The first reported by each region The regional importance score of each expert.

[0049] Furthermore, global expert importance scoring The weighted average of the regional expert importance scores reported by each region is used to obtain the overall expert importance score. The weight is the confidence weight of each region. The weighted fusion method makes the regional expert importance scores of regions with stable latency and fresh information account for a higher proportion in the overall expert importance score, thereby improving the accuracy of the overall expert importance score in a latency-fluctuating environment.

[0050] Step 5: Perform selective recalculation within the region based on the routing decision cache to generate the accurate compensation gradient for the local region. When the edge coordination node detects a failure of an agent within the local domain, it extracts the routing decision record of the failed agent's processing sample from the routing decision cache, selects a target agent with the same expert copy deployed within the local domain, and transmits the routing decision record and sample index to the target agent. The target agent performs deterministic forward propagation and gradient calculation based on the set of activated expert indexes and routing weights in the routing decision record to generate the local domain accurate compensation gradient.

[0051] It should be noted that route-consistent selective recalculation means that when the target agent performs recalculation, it no longer dynamically calculates routing decisions through a gating network, but directly uses the set of activated expert indexes stored in the routing decision record. and routing weight The sample is directly routed to a designated expert for processing, thus ensuring that the expert activation path in the recalculation is completely consistent with the original calculation.

[0052] Furthermore, the deterministic forward propagation and gradient calculation are performed as follows: after receiving the sample index, the target agent loads the corresponding sample data from local storage, and activates the expert index set in the routing decision record. Only the specified expert subnetworks are activated for forward propagation computation, and the outputs of each activated expert are calculated according to their routing weights. The final output is obtained by weighted combination, and then backpropagation is performed to calculate the gradient, generating a local accurate compensation gradient consistent with the original calculation path.

[0053] Furthermore, the weighted combination is calculated as follows: Let the first... The set of activation expert indices for each sample is The corresponding route weight is The outputs of each activation expert are as follows: ,in They represent the first The first to the second of the samples The output of each activated expert will result in the final output. Calculated as ,in Indicates traversal An index variable for activating an expert. This represents the total number of activated experts, and the final output is then used to calculate the task loss and perform backpropagation.

[0054] Furthermore, to improve recalculation efficiency, the edge coordination node prioritizes the samples processed by the faulty agent based on global expert importance scores before performing selective recalculation. Specifically, for the first sample processed by the faulty agent... For each sample, calculate its recalculation priority. : in, Indicates the first A set of activation expert indexes for each sample. This indicates traversing the set of activated expert indices. Index variables of various experts in China Experts Global expert importance score, Experts In the The routing weights corresponding to each sample routing decision. Edge coordination nodes schedule recalculation tasks in descending order of recalculation priority.

[0055] Furthermore, recalculate priorities. Taking into account the global expert importance scores and routing weights of all experts activated by the sample, the recalculation priority reflects the overall contribution of the sample to the model training. Prioritizing the recalculation of high recalculation priority samples can maximize the quality of the recovered gradient within the fault tolerance window. If it is not possible to complete the recalculation of all samples before the fault tolerance window expires, the high recalculation priority samples that have been completed first have already covered the main gradient contributions.

[0056] Furthermore, when the sample processed by the faulty agent involves experts deployed in remote regions, the following steps are also included: the edge coordination node identifies the set of experts in the remote regions involved based on the global expert importance score, and asynchronously transmits the routing decision record of the corresponding sample to the remote edge coordination node through a dedicated inter-regional channel. The remote edge coordination node coordinates the local agent to perform cross-regional route consistency recalculation, generates the remote compensation gradient, and returns the remote compensation gradient to the central coordination node.

[0057] Furthermore, the inter-regional dedicated channel refers to a dedicated network connection established between cross-regional data centers for transmitting recalculation-related control information and gradient data. The inter-regional dedicated channel adopts a priority queue and bandwidth reservation mechanism to ensure that the communication needs of the recalculation task take precedence over regular data transmission, thereby reducing the communication latency of cross-regional recalculation.

[0058] Step 6: Aggregate gradients from various sources to complete the global model parameter update. The central coordination node aggregates the local accurate compensation gradients and remote compensation gradients from various regions, performs global gradient aggregation with the gradients reported by normal agents, updates the model parameters, and completes the consistency restoration of the training process.

[0059] It should be noted that during global gradient aggregation, the central coordinating node performs a weighted average of the gradients from each source according to the number of samples corresponding to each source. Let the gradient contributed by a normal agent be... The corresponding sample size is The local precise compensation gradient is The corresponding sample size is The far-end compensation gradient is The corresponding sample size is The aggregated global gradient The calculation is as follows: Furthermore, to complete global gradient aggregation within the fault tolerance window, when remote compensation gradients fail to return within a preset time limit, the central coordination node approximates the contribution of the missing remote compensation gradients based on the global expert importance score. Specifically, an approximate compensation gradient is generated based on the average gradient statistics of remote regional experts in historical batches and the routing weight information of the current batch. Used to replace unreturned far-end compensation gradients in global gradient aggregation.

[0060] Furthermore, approximate gradient compensation The calculation method is as follows: First, extract remote regional experts from the historical window in the most recent Calculate the average gradient of each batch of gradient statistics. ,in This indicates the number of batches contained in the historical window; then, it is based on the sum of the sample routing weights involving remote regional experts in the current batch. Sum of historical average routing weights Scaling the ratio yields... ,in The value of is determined based on the stability of the model training and the gradient change rate.

[0061] Furthermore, the sum of historical average route weights This indicates that remote regional experts are in history The average of the sum of the route weights of each batch in the current batch, and the sum of the route weights of the current batch. This represents the sum of the routing weights of all samples with activated remote regional experts in the current batch, and the ratio. It reflects the change in the activation intensity of the current batch of remote regional experts relative to the historical average level. By scaling the historical average gradient with a ratio, the gradient contribution of the current batch of remote regional experts can be approximately estimated.

[0062] Furthermore, preset time limits The calculation method is determined based on the fault tolerance window duration and the time constraint of global gradient aggregation. ,in Indicates the duration of the fault tolerance window. This represents the estimated time taken for the global gradient aggregation operation. This indicates the reserved time buffer margin.

[0063] Furthermore, the estimated time consumption of the global gradient aggregation operation. This includes the total time for the central coordination node to receive gradients from various sources, perform weighted average calculations, and distribute the updated model parameters to different regions, along with the time buffer margin. To address uncertainties caused by network fluctuations and changes in computational load, global gradient aggregation is ensured to be completed within the fault-tolerant window.

[0064] This implementation obtains the mean delay and variance of each link through regional network delay state estimation. During the global expert importance score fusion process, the expert importance score vectors reported by each region are timestamped and aligned, eliminating the time distortion effect of cross-regional communication delay on expert activation statistics. At the same time, confidence weights are set based on delay variance, so that the expert importance scores of regions with high delay uncertainty receive lower weights during fusion. This maintains the accuracy of the global expert importance score in a non-stationary environment of delay fluctuation, solving the technical problem of expert contribution assessment based on outdated information.

[0065] This implementation uses a routing decision caching mechanism in the edge coordination node to pre-store the routing decision information required for selective recalculation within the region, thus shortening the critical path for recalculation from cross-regional communication to intra-regional communication when a fault occurs. Through the routing consistency selective recalculation mechanism, the target agent directly uses the routing decision cached routing decision to perform deterministic calculations, ensuring that the local accurate compensation gradient and the original expert activation path are accurately matched, thus solving the technical problem that the delay in transmitting routing decision information across regions may exceed the fault tolerance window.

[0066] This implementation adopts a hierarchical expert importance scoring mechanism. The edge coordination node first calculates the expert importance score within its region based on local domain information. The central coordination node integrates the expert importance scores within each region to generate a global expert importance score and distributes it to the edge coordination node. This enables the edge coordination node to obtain expert contribution information from a global perspective, thereby enabling it to perform expert-perceptive fault prediction and recalculation decisions. This solves the technical problem that the edge coordination node cannot independently complete the global expert contribution assessment.

[0067] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A multi-agent collaborative large model training full-process efficient fault-tolerant method, characterized in that, Includes the following steps: The central coordination node measures the network round-trip delay between the data centers in each region and the central coordination node. It uses the Kalman filter algorithm to filter the delay time series data of each link, estimates the mean delay and fluctuation variance of each link, and generates a regional network delay status estimate. During the local gradient calculation process, each agent in each region records the gated routing decision triggered by each sample, generates a routing decision record, and transmits the routing decision record to the local edge coordination node. The edge coordination node stores the received routing decision record in the routing decision cache. The edge coordination node calculates the gradient energy ratio of each expert in the current batch based on the routing decision records in the routing decision cache, and generates an expert importance score vector within the region. The central coordination node receives the time-stamped expert importance score vectors within the region reported by each edge coordination node. Based on the regional network delay state estimation, the central coordination node corrects and aligns the expert importance score vectors within each region using timestamps. Based on the delay fluctuation variance, the central coordination node sets confidence weights for each region and performs weighted fusion on the corrected expert importance score vectors within the region to generate a global expert importance score. When the edge coordination node detects a failure of an agent in the local domain, it extracts the routing decision record of the processing sample of the failed agent from the routing decision cache, selects a target agent in the local domain, and the target agent performs deterministic forward propagation and gradient calculation based on the set of activated expert indexes and routing weights in the routing decision record to generate the local domain accurate compensation gradient. The central coordination node aggregates the local precise compensation gradients and remote compensation gradients from various regions, performs global gradient aggregation with the gradients reported by normal agents, and updates the model parameters.

2. The method of claim 1, wherein, The Kalman filter algorithm employs adaptive process noise covariance, dynamically adjusting the process noise variance based on the statistical characteristics of the observation residuals. The process noise variance is obtained by multiplying the process noise variance of the previous time step by the smoothing coefficient, plus the product of the square of the observation residual at the current time step and the complement of the smoothing coefficient. The observation residual represents the difference between the actual observation delay value and the Kalman filter predicted delay value at the current time step.

3. The method of claim 1, wherein, The routing decision record includes a sample index, an active expert index set, a routing weight corresponding to each active expert, and a routing decision generation timestamp. The edge coordination node adopts a sliding window management strategy for the routing decision cache, retaining only the routing decision records of the most recent training batches. The number of batches retained is determined by dividing the fault tolerance window duration by the average processing time of a single batch and rounding up.

4. The method of claim 1, wherein, The method for calculating the regional expert importance score is as follows: the product of the activation frequency and average routing weight of each expert in the local region is divided by the sum of the products of the activation frequency and average routing weight of all experts in the local region to obtain the regional expert importance score of each expert; wherein, the average routing weight is the sum of the routing weights of all active instances of each expert in the current batch divided by the activation frequency.

5. The method of claim 4, wherein, The calculation of the regional expert importance score also incorporates the gradient norm statistics of each expert. The product of each expert's activation frequency, average routing weight, and gradient norm statistics is divided by the sum of the products of all experts in the region to obtain the regional expert importance score for each expert. The gradient norm statistics are the average L2 norm of the gradients generated by each expert in the current batch.

6. The method of claim 1, wherein, The timestamp correction and alignment process is as follows: For the expert importance score vector within each region, the original timestamp is added to the estimated mean delay value of the corresponding region to obtain the corrected equivalent timestamp, and the time deviation of each region is calculated based on the current time. The confidence weight is determined by the product of the inverse of the delay fluctuation variance and the time decay factor, where the time decay factor is an exponential function value with the natural constant as the base and the negative value of the product of the time decay coefficient and the time deviation as the exponent.

7. The method of claim 1, wherein, The fusion method for the global expert importance score is as follows: For each expert, the sum of the products of the confidence weights of the regions where the expert is deployed and the expert importance score in the corresponding region is divided by the sum of the confidence weights of the regions where the expert is deployed to obtain the global expert importance score for that expert.

8. The method of claim 1, wherein, The target agent performs deterministic forward propagation and gradient calculation as follows: based on the set of activated expert indexes in the routing decision record, only the specified expert subnetwork is activated for forward propagation calculation. The outputs of each activated expert are weighted and combined according to the routing weights to obtain the final output. Then, backpropagation is performed to calculate the gradient. The weighted combination is the sum of the products of the outputs of each activated expert and their corresponding routing weights.

9. The method of claim 1, wherein, Before performing selective recalculation, the edge coordination node prioritizes the samples processed by the faulty agent based on the global expert importance score. The recalculation priority of each sample is the sum of the products of the global expert importance scores of all experts activated by the sample and the corresponding routing weights. The recalculation tasks are scheduled in descending order of recalculation priority.

10. A multi-agent collaborative large model training full-process efficient fault-tolerant system for executing the method of any one of claims 1 to 9. include: The delay state estimation module is used to measure the network round-trip delay between regional data centers and the central coordination node. It uses the Kalman filter algorithm to filter the delay time series data of each link, estimates the mean delay and fluctuation variance of each link, and generates a regional network delay state estimate. The routing decision cache module is used to record the gated routing decisions triggered by each sample during the local gradient calculation process of the agent in each region, generate routing decision records and store them in the routing decision cache; The regional scoring calculation module is used to calculate the activation frequency and cumulative routing weight of each expert in the current batch based on the routing decision records in the routing decision cache, calculate the gradient energy ratio of each expert, and generate a regional expert importance scoring vector. The delay compensation fusion module is used to receive the time-stamped regional expert importance score vectors reported by each edge coordination node, perform timestamp correction and alignment based on the regional network delay state estimation, set confidence weights based on delay fluctuation variance, and perform weighted fusion on the corrected regional expert importance score vectors to generate a global expert importance score. The route consistency recalculation module is used to extract the route decision records of the failed agent's processing samples from the route decision cache when a failure of an agent in the local domain is detected. It selects the target agent and performs deterministic forward propagation and gradient calculation based on the set of activated expert indexes and route weights in the route decision records to generate accurate compensation gradients for the local domain. The global gradient aggregation module is used to summarize the local accurate compensation gradients and remote compensation gradients from various regions, and perform global gradient aggregation with the gradients reported by normal agents to update the model parameters.

Images