A delay-aware based adaptive distributed training method, device and medium

Abstract

The application discloses a delay-aware-based adaptive distributed training method and device and a medium, and relates to the technical field of distributed training. The method comprises the following steps: monitoring network state data of nodes in a distributed cluster, wherein the network state data comprises inter-node communication delay, resource utilization rate and data packet loss rate; constructing a network quality evaluation model based on the network state data, and generating a network quality score; using a time series prediction model to perform multi-step prediction on the communication delay, and establishing delay gradient compensation in combination with historical gradient transmission data; adjusting training parameters by using a reinforcement learning framework according to the network state data and the prediction result, and adopting time-effectiveness weighted gradient aggregation, and adjusting aggregation weights according to gradient age and node delay. The application realizes intelligent training process optimization in different network environments by the above method, and is suitable for cross-data-center training and edge computing collaborative training scenarios.

Description

Technical Field

[0001] This application relates to the field of distributed training technology, and in particular to a latency-aware adaptive distributed training method, device and medium. Background Technology

[0002] As the scale of deep learning models continues to expand, distributed training has become a key technology for accelerating model convergence. Distributed training methods are mainly divided into two categories: synchronous training and asynchronous training. Synchronous training ensures consistency through global gradient synchronization, but is limited by the performance of the slowest node. Asynchronous training allows nodes to update independently, reducing waiting time, but may cause model oscillation or unstable convergence due to outdated gradients.

[0003] Existing asynchronous training optimization methods still suffer from the following problems: Static policies have poor adaptability; most methods cannot dynamically adjust to network latency fluctuations, leading to a sharp decline in training efficiency in high-latency environments. Gradient compensation has limitations; existing latency compensation algorithms assume a fixed latency distribution, making it difficult to cope with sudden network congestion or instability in cross-regional training. Resource utilization is uneven: traditional methods cannot dynamically adjust batch size or learning rate based on real-time computing load and network status, resulting in insufficient GPU utilization or exacerbated gradient conflicts. Existing solutions lack convergence proofs in dynamic network environments, especially in large-scale heterogeneous clusters where stability is difficult to guarantee.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: Existing asynchronous distributed training methods suffer from low training efficiency, unstable model convergence, and uneven resource allocation in scenarios involving network latency fluctuations, heterogeneous computing environments, and dynamic resource competition. Summary of the Invention

[0005] This application provides a latency-aware adaptive distributed training method, device, and medium that can solve the problems of low training efficiency, unstable model convergence, and uneven resource allocation in existing asynchronous distributed training methods under network latency fluctuations, heterogeneous computing environments, and dynamic resource competition scenarios.

[0006] In a first aspect, embodiments of this application provide a latency-aware adaptive distributed training method, the method comprising: monitoring network status data of nodes in a distributed cluster, the network status data including inter-node communication latency, resource utilization, and packet loss rate; constructing a network quality assessment model based on the network status data to generate a network quality score; using a time series prediction model to perform multi-step prediction of communication latency, and establishing latency gradient compensation by combining historical gradient transmission data; adjusting training parameters using a reinforcement learning framework based on the network status data and prediction results, and employing time-sensitive weighted gradient aggregation, adjusting the aggregation weights according to gradient age and node latency, the training parameters including gradient waiting threshold, learning rate scaling factor, and batch size adjustment factor.

[0007] In one implementation of this application, a network quality assessment model is constructed based on network state data to generate a network quality score. Specifically, this includes: abstracting the distributed cluster into a graph structure, where nodes represent computing units and edges represent communication links; using a graph convolutional network to encode the features of communication latency, bandwidth utilization, and data packet loss rate to generate a topology embedding vector; calculating the quality score based on the temporal correlation between the topology embedding vector and historical network state data; and adjusting the monitoring priority and sampling frequency of communication links based on the quality score.

[0008] In one implementation of this application, a time series prediction model is used to perform multi-step prediction of communication delay, and delay gradient compensation is established by combining historical gradient transmission data. Specifically, this includes: using a long short-term memory network combined with an attention mechanism to model the communication delay and output multi-step delay prediction results; calculating a timeliness attenuation coefficient based on the deviation between the multi-step delay prediction results and the real-time delay; compensating for gradients with deviations exceeding a threshold based on the timeliness attenuation coefficient, and smoothly updating the remaining gradients using historical compensation parameters.

[0009] In one implementation of this application, training parameters are adjusted using a reinforcement learning framework based on network state data and prediction results. Specifically, this includes: constructing a policy controller with an Actor-Critic architecture, inputting network quality scores and resource utilization; and outputting gradient waiting threshold levels and learning rate scaling factors through the discrete action space of the policy controller.

[0010] In one implementation of this application, the method further includes: calculating a resource efficiency factor by combining network quality score and resource utilization rate; monitoring the temporal change rate of gradient conflict degree as a stability indicator; and weightedly fusing the resource efficiency factor and stability indicator to generate a composite reward signal to drive the policy gradient update of the Actor-Critic network.

[0011] In one implementation of this application, time-weighted gradient aggregation is adopted, and the aggregation weight is adjusted according to the gradient age and node delay. Specifically, it includes: calculating a time-sensitivity factor based on the prediction delay deviation value and the gradient age; adjusting the gradient aggregation weight through the time-sensitivity factor so that the gradients of low-time-sensitivity nodes are reduced in weight during aggregation; and feeding the adjusted aggregation weight back to the policy controller to correct the decision boundary of the batch size adjustment factor.

[0012] In one implementation of this application, dynamic task scheduling is also included: calculating the communication cost matrix between nodes based on the topology embedding vector, generating training subtask allocation priorities; when the node resource utilization rate is detected to exceed the threshold, the untrained data batch is reallocated according to the batch size adjustment factor; the gradient of the high communication cost link is sparsely compressed, and the compression ratio is positively correlated with the prediction latency.

[0013] In one implementation of this application, after calculating the timeliness attenuation coefficient based on the deviation between the multi-step delay prediction result and the real-time delay, the method further includes: pausing the gradient update of the problem node when the prediction deviation value continuously exceeds a threshold, and reconstructing its network state estimate using the topological embedding vector; generating a compensation training batch according to the policy controller to accelerate the recovery of node parameter synchronization; and reactivating the training task after the quality score of the node network recovers to a safe threshold.

[0014] Secondly, embodiments of this application also provide a latency-aware adaptive distributed training device, the device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: monitor network status data of nodes in a distributed cluster, the network status data including inter-node communication latency, resource utilization, and packet loss rate; construct a network quality assessment model based on the network status data to generate a network quality score; perform multi-step prediction of communication latency using a time series prediction model, and establish latency gradient compensation by combining historical gradient transmission data; adjust training parameters using a reinforcement learning framework according to the network status data and prediction results, and adopt time-sensitive weighted gradient aggregation, adjusting the aggregation weights according to gradient age and node latency, the training parameters including gradient waiting threshold, learning rate scaling factor, and batch size adjustment factor.

[0015] Thirdly, embodiments of this application also provide a latency-aware adaptive distributed training non-volatile computer storage medium storing computer-executable instructions. These instructions are configured to: monitor network status data of nodes in a distributed cluster, including inter-node communication latency, resource utilization, and packet loss rate; construct a network quality assessment model based on the network status data to generate a network quality score; perform multi-step prediction of communication latency using a time-series prediction model and establish latency gradient compensation by combining historical gradient transmission data; adjust training parameters using a reinforcement learning framework based on the network status data and prediction results, and employ time-sensitive weighted gradient aggregation, adjusting the aggregation weights according to gradient age and node latency. The training parameters include a gradient waiting threshold, a learning rate scaling factor, and a batch size adjustment factor.

[0016] This application provides a latency-aware adaptive distributed training method, device, and medium. By combining real-time network quality assessment and latency prediction with reinforcement learning to dynamically adjust training parameters, it significantly improves training efficiency in network fluctuation scenarios. A latency compensation mechanism based on priority queues corrects only significant deviation gradients, reducing computational overhead and mitigating gradient staleness. A topology-aware resource scheduling strategy and time-weighted aggregation are introduced to balance computational load and communication costs, avoiding resource waste. A composite reward signal is used to fuse resource efficiency and gradient conflict to drive policy optimization, reducing model oscillations and ensuring convergence in dynamic environments. Combining graph convolutional networks and a meta-learning optimizer enables policy transfer across heterogeneous nodes and network topologies, improving system generalization capabilities. Through intelligent network state monitoring, dynamic policy adjustment, and gradient optimization techniques, it provides an efficient, stable, and highly adaptable distributed training solution. This system can achieve intelligent training process optimization in different network environments and is widely applicable to scenarios such as cross-data center training and edge computing collaborative training, providing innovative technical support for improving the performance of large-scale distributed machine learning systems. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a latency-aware adaptive distributed training method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the internal structure of a latency-aware adaptive distributed training device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a latency-aware adaptive distributed training method, device, and medium, which solves the problems of low training efficiency, unstable model convergence, and uneven resource allocation in the asynchronous distributed training methods of the prior art under network latency fluctuations, heterogeneous computing environments, and dynamic resource competition scenarios.

[0020] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 This document provides a flowchart of a latency-aware adaptive distributed training method as an embodiment of this application. Figure 1 As shown in the embodiment of this application, an adaptive distributed training method based on latency awareness is provided, which specifically includes the following steps: Step 10: Monitor the network status data of nodes in the distributed cluster. The network status data includes inter-node communication latency, resource utilization, and packet loss rate. In this step, a lightweight probe is deployed on each computing node to measure the bidirectional communication latency between nodes via UDP probe packets, which carry nanosecond-level timestamps.

[0022] Step 20: Construct a network quality assessment model based on network state data and generate a network quality score; As an optional implementation, a network quality assessment model is constructed based on network state data to generate a network quality score. Specifically, this may include: Step 201: Abstracting the distributed cluster into a graph structure, where nodes represent computing units and edges represent communication links; Step 202: Using a graph convolutional network to encode the features of communication latency, bandwidth utilization, and data packet loss rate to generate a topology embedding vector; Step 203: Calculating the quality score based on the temporal correlation between the topology embedding vector and historical network state data; Step 204: Adjusting the monitoring priority and sampling frequency of communication links based on the quality score.

[0023] In this step, vertices represent physical / pseudo-computing nodes, with each vertex recording real-time metrics such as CPU utilization and memory usage. Edges represent communication links between nodes, with each edge storing communication latency, bandwidth utilization, and packet loss rate. A graph convolutional network is used for feature extraction. Each node collects latency, bandwidth, and packet loss rate data from its directly connected links. An initial embedding vector is generated through a fully connected layer, and two layers of information transfer are performed: the first layer aggregates resource utilization features from adjacent nodes, and the second layer fuses communication quality metrics across hops, generating a 128-dimensional topology embedding vector. Finally, mean pooling is applied to the embeddings of all nodes in the graph to output a topology fingerprint representing the overall network state. Inputting a topology fingerprint sequence from a preset time period, a temporal convolutional network is used to extract temporal patterns and identify key features.

[0024] Step 30: Use a time series prediction model to make multi-step predictions of communication delays, and combine historical gradient transmission data to establish delay gradient compensation; As an optional embodiment, a time series prediction model is used to predict communication delay in multiple steps, and delay gradient compensation is established by combining historical gradient transmission data. Specifically, it may include: Step 301: Using a long short-term memory network combined with an attention mechanism to model the communication delay and output the multi-step delay prediction results; Step 302: Calculating the timeliness attenuation coefficient based on the deviation between the multi-step delay prediction results and the real-time delay; Step 303: Compensating the gradient with deviation exceeding the threshold based on the timeliness attenuation coefficient, and updating the remaining gradient smoothly using historical compensation parameters.

[0025] As an optional embodiment, and employing time-sensitive weighted gradient aggregation, the aggregation weights are adjusted based on gradient age and node latency. Specifically, this may include: Step 304: Calculating a time-sensitive factor based on the prediction latency deviation value and gradient age; Step 305: Adjusting the gradient aggregation weights through the time-sensitive factor so that the gradients of low-time-sensitive nodes are reduced in weight during aggregation; Step 306: Feeding the adjusted aggregation weights back to the policy controller to correct the decision boundary of the batch size adjustment factor.

[0026] In this step, compared to traditional time-series models such as ARIMA, LSTM with attention mechanism can capture long-term dependencies, while the attention mechanism further enhances the identification of key time points, such as moments of network congestion. Dynamic gradient compensation proposes importance-aware compensation, correcting only the preset top gradients, avoiding the computational overhead of global compensation, whereas traditional methods require compensating all gradients. Through time-series prediction and dynamic compensation mechanisms, this step can further alleviate the gradient staleness problem in asynchronous training. Specifically, an LSTM network is used to model delayed time-series features, with input being delayed observations from the past 30 epochs, and outputting predictions for the next 5 time steps. The network structure consists of a 3-layer, 128-unit LSTM, with an attention mechanism added to capture sudden delays at key time points, and a timeliness decay function designed to dynamically adjust the weights of outdated gradients. For the gradient of the delay Δt, the compensation coefficient is λ=1 / (1+exp(-(αΔt+βσ²)), where σ² is the gradient variance, and α and β are learnable parameters. When the deviation between the predicted delay and the actual delay exceeds a threshold, an emergency mode is triggered, pausing gradient updates and re-evaluating the network state to prevent model divergence caused by incorrect compensation.

[0027] Step 40: Based on the network state data and prediction results, adjust the training parameters using the reinforcement learning framework and adopt time-sensitive weighted gradient aggregation. Adjust the aggregation weights according to the gradient age and node latency. The training parameters include gradient waiting threshold, learning rate scaling factor and batch size adjustment factor.

[0028] As an optional embodiment, the training parameters are adjusted using a reinforcement learning framework based on network state data and prediction results. Specifically, this may include: Step 401: Constructing a policy controller with an Actor-Critic architecture, inputting network quality score and resource utilization; Step 402: Outputting gradient waiting threshold level and learning rate scaling factor through the discrete action space of the policy controller.

[0029] As an optional embodiment, the method may further include: step 403: calculating a resource efficiency factor by combining network quality score and resource utilization rate; step 404: monitoring the temporal change rate of gradient conflict degree as a stability index; step 405: weighted fusion of resource efficiency factor and stability index to generate a composite reward signal to drive policy gradient update of Actor-Critic network.

[0030] In this step, discrete and continuous actions are modeled uniformly. Reinforcement learning is used to automatically explore the optimal combination, employing a resource-aware reward function. Traditional RL only optimizes convergence speed; this step incorporates resource consumption penalties, such as GPU utilization, to achieve Pareto optimality. A reinforcement learning-based policy controller dynamically optimizes training parameters. Ten-dimensional metrics, including network QoS score, gradient conflict degree, and resource utilization, are input into the policy network. An Actor-Critic architecture is used, with Actors outputting discrete and continuous actions, i.e., learning rate scaling factors. A distributed PPO algorithm is used, with eight workers collecting environmental interaction data in parallel, updating the policy network every 1000 steps. The reward function integrates training speed, model accuracy, and resource consumption, with the formula R = 0.6ΔAcc + 0.3ΔSpeed ​​− 0.1Resource. Dynamically adjusted parameters are distributed to each node via a control bus to ensure global policy consistency. Experimental results show that this module reduces convergence time by 28% in ResNet-50 training.

[0031] Furthermore, weights are dynamically adjusted based on gradient age and node latency to maximize the contribution of fresh gradients. When node latency is too high, its batch size is automatically reduced and a backup node is triggered, reducing recovery time compared to traditional fault-tolerance schemes. Efficient gradient synchronization and resource management are achieved through elastic aggregation: aggregation weights are dynamically adjusted based on node latency, the Paxos protocol is used for node failure recovery, model checkpoints are saved every epoch, and the state is synchronized from the majority replicas after a failed node restarts. When node latency exceeds a threshold, its batch size is automatically reduced and a backup computing unit is activated to ensure training continuity.

[0032] As an optional embodiment, the method may further include: calculating the communication cost matrix between nodes based on the topology embedding vector, generating a training subtask allocation priority; when the node resource utilization is detected to exceed a threshold, reallocating untrained data batches according to the batch size adjustment factor; and sparsely compressing the gradient of high communication cost links, with the compression ratio being positively correlated with the prediction latency.

[0033] In this step, based on the similarity between nodes calculated by topological embedding, model shards that require frequent synchronization are preferentially allocated to low-cost node pairs. For high-cost links, the amount of data transmitted in a single transmission is reduced, gradient sparsity compression is enabled, and when the CPU utilization of a node exceeds a threshold for a continuous preset time, some of the data to be processed is migrated to the neighboring node with the lowest communication cost, automatically reducing the batch size of that node, and additional data shards are dynamically allocated to idle nodes to improve the overall utilization of the cluster.

[0034] As an optional embodiment, after calculating the timeliness attenuation coefficient based on the deviation between the multi-step delay prediction result and the real-time delay, the method may further include: pausing the gradient update of the problem node when the prediction deviation value continuously exceeds a threshold, and reconstructing its network state estimate using the topological embedding vector; generating a compensation training batch according to the policy controller to accelerate the recovery of node parameter synchronization; and reactivating the training task after the quality score of the node network recovers to a safe threshold.

[0035] In this step, parameter updates for a node are paused, its last effective model version is recorded, and the estimated state of that version is reconstructed using topological embeddings of neighboring nodes, generating temporary virtual gradients. Data batches not processed during the failure period are split into micro-batches of more than twice the normal size for compensatory training. Initially, the batch size of the aforementioned versions for each node is limited to half or less of the normal value. After a preset period without anomalies, the limit is lifted, and complete data shards are restored. The latest parameters are pulled from the nearest neighbor node, a weighted average is calculated, and then synchronized to that node. The loss value after compensatory training is compared to confirm that the node's model accuracy has recovered to the pre-failure level.

[0036] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a latency-aware adaptive distributed training device, the structure of which is as follows: Figure 2 As shown.

[0037] Figure 2 This is a schematic diagram of the internal structure of a latency-aware adaptive distributed training device provided in an embodiment of this application. Figure 2 As shown, the device includes: At least one processor 201; And a memory 202 that is communicatively connected to at least one processor; The memory 202 stores instructions executable by at least one processor 201, which enables the processor 201 to: monitor network status data of nodes in a distributed cluster, including inter-node communication latency, resource utilization, and packet loss rate; construct a network quality assessment model based on the network status data to generate a network quality score; perform multi-step prediction of communication latency using a time series prediction model and establish latency gradient compensation by combining historical gradient transmission data; and adjust training parameters using a reinforcement learning framework based on the network status data and prediction results, and adopt time-weighted gradient aggregation, adjusting the aggregation weights according to gradient age and node latency, with training parameters including gradient waiting threshold, learning rate scaling factor, and batch size adjustment factor.

[0038] Some embodiments of this application provide corresponding to Figure 1A non-volatile computer storage medium for latency-aware adaptive distributed training stores computer-executable instructions. These instructions are configured to: monitor network state data of nodes in a distributed cluster, including inter-node communication latency, resource utilization, and packet loss rate; construct a network quality assessment model based on the network state data to generate a network quality score; perform multi-step prediction of communication latency using a time-series prediction model and establish latency gradient compensation by combining historical gradient transmission data; and adjust training parameters using a reinforcement learning framework based on the network state data and prediction results, employing time-sensitive weighted gradient aggregation, adjusting the aggregation weights according to gradient age and node latency. The training parameters include a gradient waiting threshold, a learning rate scaling factor, and a batch size adjustment factor.

[0039] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0040] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0041] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0042] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0045] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0046] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0047] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0048] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0049] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A latency-aware adaptive distributed training method, characterized in that, The method includes: Monitor network status data of nodes in a distributed cluster, including inter-node communication latency, resource utilization, and packet loss rate; Based on the network state data, a network quality assessment model is constructed to generate a network quality score, specifically including: The distributed cluster is abstracted as a graph structure, where nodes represent computing units and edges represent communication links; The communication delay, bandwidth utilization, and data packet loss rate are feature-encoded using a graph convolutional network to generate a topological embedding vector. The quality score is calculated based on the temporal correlation between the topology embedding vector and the historical network state data; The monitoring priority and sampling frequency of the communication link are adjusted based on the quality score. The communication delay is predicted in multiple steps using a time series prediction model, and delay gradient compensation is established by combining historical gradient transmission data. Specifically, this includes: The communication delay is modeled using a long short-term memory network combined with an attention mechanism, and multi-step delay prediction results are output. The timeliness attenuation coefficient is calculated based on the deviation between the multi-step delay prediction result and the real-time delay. The gradient with deviation value exceeding the threshold is compensated based on the time-sensitivity decay coefficient, and the remaining gradient is smoothly updated using historical compensation parameters. Based on the network state data and prediction results, training parameters are adjusted using a reinforcement learning framework, and time-weighted gradient aggregation is employed. The aggregation weights are adjusted according to gradient age and node latency. The training parameters include a gradient waiting threshold, a learning rate scaling factor, and a batch size adjustment factor, specifically including: Constructing Actors The policy controller of the Critic architecture takes the network quality score and resource utilization rate as input. The policy controller outputs gradient waiting threshold levels and learning rate scaling factors through the discrete action space of the policy controller. The timeliness factor is calculated based on the deviation value and the gradient age. The aggregation weights are adjusted by the timeliness factor so that the gradients of low-timeliness nodes are reduced in weight during aggregation. The adjusted aggregation weights are fed back to the policy controller to correct the decision boundary of the batch size adjustment factor.

2. The latency-aware adaptive distributed training method according to claim 1, characterized in that, The method further includes: The resource efficiency factor is calculated by combining the network quality score and resource utilization rate. The temporal rate of change of gradient conflict degree is monitored as a stability indicator; The resource efficiency factor and stability index are weighted and fused to generate a composite reward signal, which drives the Actor. Policy gradient update of the Critic network.

3. The latency-aware adaptive distributed training method according to claim 1, characterized in that, The method further includes: Based on the topology embedding vector, calculate the communication cost matrix between nodes and generate training subtask allocation priorities; When the node resource utilization rate is detected to exceed the threshold, the untrained data batch is reallocated according to the batch size adjustment factor. The gradient of the high-cost communication link is sparsified and compressed, and the compression ratio is positively correlated with the multi-step delay prediction result.

4. The latency-aware adaptive distributed training method according to claim 1, characterized in that, After calculating the timeliness attenuation coefficient based on the deviation between the multi-step delay prediction result and the real-time delay, the method further includes: When the predicted deviation value continuously exceeds the threshold, the gradient update of the problem node is paused, and the network state estimate is reconstructed using the topological embedding vector. The strategy controller generates compensation training batches to accelerate parameter synchronization of recovery nodes; Once the quality score of the node network recovers to a safe threshold, the training task is reactivated.

5. A latency-aware adaptive distributed training device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Perform the steps of the latency-aware adaptive distributed training method as described in any one of claims 1-4.

6. A non-volatile computer storage medium based on delay-aware adaptive distributed training, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Perform the steps of the latency-aware adaptive distributed training method as described in any one of claims 1-4.

Images