A method and apparatus for selecting recomputation timing during large model training

Abstract

This invention discloses a method and apparatus for selecting recomputation timing during large model training, belonging to the field of distributed training optimization technology for large models at the intersection of artificial intelligence and high-performance computing. The method includes: dynamically predicting the impact of pipeline bubbles and memory usage by real-time monitoring of computation, communication tasks, and memory usage during training iterations, using a pre-trained machine learning model, and determining the timing for inserting recomputation operations. This method takes large model training state data as input, aims to minimize iteration latency, and intelligently outputs the optimal recomputation trigger timing. The apparatus includes state monitoring, intelligent decision-making, feature engineering processing, and execution modules to implement this method. This invention upgrades static rules to intelligent decision-making, enabling more precise utilization of computation and communication gaps, adaptively balancing computation, communication, and memory resources, thereby significantly improving the throughput and efficiency of large model training.

Description

Technical Field

[0001] This invention relates to distributed training optimization technology for large models in the intersection of artificial intelligence and high-performance computing, and particularly to an intelligent decision-making method and supporting device based on machine learning for dynamically selecting the timing of activation value recomputation operations during training. Specifically, it relates to a method and device for selecting the timing of recomputation in large model training. Background Technology

[0002] In recent years, with the popularization of the Transformer architecture and the rapid development of computing infrastructure, the scale of deep learning models (including the number of parameters, layers, and context length) has shown an exponential growth trend, officially entering the era of "large models" and even "giant models." Large models, represented by GPT-3, PaLM, and LLaMA, have made breakthrough progress in tasks such as natural language processing, multimodal understanding, and code generation. However, these behemoths with hundreds of billions or even trillions of parameters pose unprecedented challenges to hardware resources during their training process.

[0003] First, from a computational perspective, the number of floating-point operations (FLOPs) required for a single forward / backward propagation is extremely large, necessitating the use of supercomputing clusters composed of hundreds or thousands of GPUs or TPUs for distributed acceleration through hybrid parallelism strategies such as tensor parallelism, data parallelism, and pipeline parallelism. Pipeline parallelism, in particular, divides the model vertically into multiple stages and distributes them across different devices, utilizing micro-batch technology to form pipelined jobs, aiming to improve hardware utilization. However, an inherent drawback of pipeline parallelism is the introduction of "pipeline bubbles." When a device completes the computation of its micro-batch, if the data for the next micro-batch has not yet arrived from the previous stage, the device must enter an idle waiting state; this period is the bubble. The existence of bubbles directly reduces the effective computation time of the devices, becoming one of the key factors limiting training throughput.

[0004] Secondly, from a memory perspective, the biggest bottleneck in training large models is often not computational power, but rather the VRAM capacity of the device (especially GPUs). In the standard backpropagation algorithm, all intermediate activations generated during forward propagation must be completely stored in VRAM until the corresponding backpropagation computation is completed. For deep networks with dozens or even hundreds of layers, the VRAM space occupied by these activations often far exceeds the model parameters themselves, becoming a "memory wall" that limits the size of trainable models. To overcome this limitation, "activation recomputation," also known as "checkpointing," has emerged. The core idea of ​​this technique is to "trade time for space": during forward propagation, only a subset of activations from key layers are selectively saved as "checkpoints," while the remaining intermediate results are discarded; when backpropagation requires the activations of a certain layer, if the value has not been saved, a new forward computation is performed starting from the nearest upstream checkpoint to recover it. This method can significantly reduce peak memory usage (typically by as much as 60%-70%), but at the cost of increased computational overhead, which in turn prolongs the total time for a single iteration.

[0005] Currently, most recomputation strategies adopted in industry and academia are static and predefined. For example, in mainstream frameworks such as Megatron-LM, users typically need to configure the recomputation scheme based on experience or simple heuristics (such as setting a checkpoint every N layers) before starting the training script. This static strategy has two fundamental drawbacks:

[0006] First, it lacks runtime adaptability. The actual training environment for large models is highly dynamic and unpredictable. The actual time consumption of computational tasks (such as the execution time of different CUDA kernels) and communication tasks (such as AllReduce, P2P Send / Recv) is affected by a variety of factors, including but not limited to: GPU instantaneous load, PCIe / NVLink bandwidth fluctuations, network congestion, and operating system scheduling latency. A static recomputation point designed in an ideal environment may happen to be inserted into a valuable computation-communication overlap window during real-world runtime, thereby disrupting the original parallel optimization. This not only fails to save time but may also exacerbate pipeline bubbles by disrupting the efficient pipeline, resulting in more harm than good.

[0007] Second, static strategies fail to comprehensively balance the complex relationship between memory and time. They typically focus only on maximizing memory savings, neglecting the overall impact of recomputation on overall iteration latency. For example, forcibly triggering recomputation at a time when memory pressure is low may save a small amount of memory but unnecessarily increases the computational burden. Conversely, at a critical point where memory is about to overflow, precisely utilizing an upcoming, relatively long communication wait interval (i.e., a bubble) to perform recomputation can achieve "zero-cost" or "low-cost" memory reclamation, thereby avoiding training crashes or sharp performance degradation due to insufficient memory.

[0008] Therefore, there is an urgent need in this field for a dynamic optimization method that can sense the real-time state of the system and intelligently decide when to trigger recomputation. This method should be able to precisely embed the "double-edged sword" operation of recomputation into the most suitable "gaps" in the training process, thereby minimizing its negative impact on training speed while ensuring memory safety, and even achieving net benefits through clever scheduling in some scenarios. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the present invention aims to provide a method and apparatus for selecting recalculation timing during large model training.

[0010] To achieve the above objectives, the technical solution adopted by this invention is as follows: a method for selecting the timing of recalculation during large model training, comprising the following steps:

[0011] Real-time monitoring of system status parameters during large model training process, including computation task execution progress, communication task execution progress, device memory usage, and pipeline bubbles generated by the overlapping relationship between computation and communication tasks;

[0012] The system state parameters are subjected to feature engineering to select the key feature set that has the highest correlation with the performance of recomputation operations;

[0013] The key feature set is input into a pre-trained machine learning decision model. The machine learning decision model comprehensively analyzes the spatiotemporal characteristics of the pipeline bubble, the current and predicted memory constraints, and outputs the candidate time window for recomputation and the corresponding prediction delay.

[0014] Select the candidate time window with the shortest prediction delay as the optimal timing;

[0015] The recalculation operation is triggered at the optimal time.

[0016] Furthermore, the real-time monitoring of system state parameters during the large model training process includes: integrating a performance profiling agent into the training framework's underlying or driver layer to achieve real-time monitoring of the system state parameters.

[0017] Furthermore, the feature engineering process includes:

[0018] Based on the underlying indices of the system state parameters, composite features of different system state parameters are constructed using domain knowledge;

[0019] From the composite features, the composite features most correlated with the target variable of the iterative delay increment introduced by recalculation are selected as key features to obtain the initial key feature set;

[0020] The numerical features in the initial key feature set are standardized and scaled, and the categorical features are one-hot encoded to obtain the key feature set.

[0021] Furthermore, the pre-trained machine learning decision model is obtained through the following offline training process:

[0022] Run multiple samples that include recomputation operations, configure different recomputation timing selection strategies for the core data objects in each training sample, collect the corresponding system state parameters as training features, and record the training iteration delay under each strategy.

[0023] Each training sample is labeled with an optimal policy label to form a training feature set with the optimal policy label. The optimal policy label is the recomputation timing selection policy that can achieve the shortest training iteration delay in the configuration of the sample.

[0024] Based on the training feature set with the optimal policy label, a machine learning decision model is trained using a supervised learning algorithm to obtain a pre-trained machine learning decision model.

[0025] Furthermore, the comprehensive analysis includes:

[0026] Evaluate whether maximizing computational communication overlap will cause a memory overflow; if a memory overflow occurs, reduce the number of input samples or initiate recomputation.

[0027] Predict how the increased computation time after inserting a recalculation operation will change the original computation and communication overlap relationship, and whether new pipeline bubbles will be generated or existing bubbles will be expanded. If new pipeline bubbles are generated, the timing of adding recalculation will be reconsidered.

[0028] Furthermore, the selection of the candidate time window with the shortest prediction latency as the optimal timing includes: using memory bottlenecks caused by recalculation or expanding existing pipeline bubbles within the candidate time window as evaluation indicators for prediction latency.

[0029] Furthermore, triggering the recalculation operation at the optimal time includes: inserting the recalculation operation into the code position corresponding to the optimal time through source-to-source or just-in-time compilation technology.

[0030] Furthermore, the algorithm used in the machine learning decision model is one of random forest, decision tree, random tree, bagging algorithm or REPTree.

[0031] The present invention also provides a device for selecting the timing of recomputation during large model training, comprising:

[0032] The status monitoring module is used to monitor the system status parameters in real time during the training process of large models. The system status parameters include the execution progress of computing tasks, the execution progress of communication tasks, the device memory usage, and pipeline bubbles generated by the overlap between computing and communication tasks.

[0033] The feature engineering processing module is used to perform feature engineering processing on the system state parameters and filter out the key feature set that is most correlated with the performance impact of recomputation operations.

[0034] The intelligent decision-making module, based on the key feature set, comprehensively analyzes the spatiotemporal characteristics of the production line bubbles, current and predicted memory limitations, through a machine learning decision model, and outputs candidate time windows for recalculation and corresponding prediction delays; the candidate time window with the shortest prediction delay is selected as the optimal timing.

[0035] The execution module triggers a recalculation operation at the optimal time.

[0036] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the method described thereon.

[0037] The beneficial effects of this invention are as follows:

[0038] Significantly improves training efficiency: By precisely scheduling recomputation operations in the idle gaps (bubbles) of computation or communication, interference with the critical path is effectively avoided, thereby saving memory and minimizing additional computational overhead, thus improving the overall training throughput.

[0039] Enhanced system robustness: Dynamically senses memory pressure and can proactively and safely trigger recalculation before memory reaches its limit, effectively preventing training interruptions caused by Out-of-Memory (OOM) and improving the stability of large-scale training tasks.

[0040] Achieving resource synergistic optimization: This method incorporates the three core resources of computing, communication, and memory into a unified optimization framework, achieving an adaptive balance among them, and surpassing the limitations of traditional methods that only focus on a single dimension (such as memory only or computing only).

[0041] High versatility: As a general scheduling strategy, this invention can be seamlessly integrated into existing mainstream deep learning frameworks (such as PyTorch and TensorFlow) and distributed training libraries (such as DeepSpeed ​​and Megatron-LM), and is suitable for various large model architectures based on pipeline parallelism. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the method of the present invention;

[0044] Figure 2 This is a schematic diagram of the closed-loop control system of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0046] This invention provides a method for selecting recomputation timing during large model training. The core of this invention lies in constructing a closed-loop control system composed of three parts: a "state monitor," an "intelligent decision engine," and a "dynamic executor." Figure 2 As shown, the status monitor is responsible for collecting system status in real time with low overhead; the intelligent decision engine (i.e., the machine learning decision model) is responsible for predicting and evaluating the advantages and disadvantages of different recalculation timings based on historical experience and the current state; and the dynamic executor is responsible for translating the decision results into specific code operations at runtime.

[0047] like Figure 1 As shown, the method of the present invention specifically includes the following steps:

[0048] Step S100: Monitor the system status parameters during the large model training process in real time. The system status parameters include the progress of computation tasks, the progress of communication tasks, the amount of device memory used, and pipeline bubbles generated by the overlap between computation and communication tasks.

[0049] Step S200: Perform feature engineering on the system state parameters to select the key feature set that has the highest correlation with the performance of recomputation operations.

[0050] Step S300: Input the key feature set into the pre-trained machine learning decision model, and use the machine learning decision model to comprehensively analyze the spatiotemporal characteristics of the pipeline bubble, the current and predicted memory limitations, and output the candidate time window for recalculation and the corresponding prediction delay.

[0051] Step S400: Select the candidate time window with the shortest prediction delay as the optimal timing.

[0052] Step S500: Trigger the recalculation operation at the optimal time.

[0053] In a preferred embodiment, step S100 forms the basis for intelligent decision-making, and the real-time monitoring process is specifically implemented in the following ways:

[0054] Integrate a lightweight, high-fidelity performance profiling agent at the bottom layer or driver layer of the training framework. This agent continuously tracks the following key state parameters with extremely low performance overhead:

[0055] Computation task execution progress: This includes not only the start / end timestamps of the GPU kernel, the active cycle and utilization of the streaming multiprocessor (SM), but also the time distribution of different computation types (such as matrix multiplication, LayerNorm, Softmax) recorded in a fine-grained manner.

[0056] Communication task execution progress: accurately capture the initiation, data transmission, and completion events of various collective communications (such as AllReduce, AllGather) and point-to-point communications (such as Send, Recv), and record their bandwidth utilization and queuing latency.

[0057] Device memory usage: Real-time monitoring of total GPU memory and host memory usage, usage, peak usage, and memory allocation / release events. Pay particular attention to memory pool changes related to activation values, gradients, and optimizer states.

[0058] Pipeline Bubbles: By analyzing the intersection of timelines of events and communication events, idle time windows (i.e., bubbles) existing on various devices are intelligently identified and quantified. This includes the bubble's start time, duration, and cause (e.g., waiting for upstream data, waiting for downstream to be ready).

[0059] All monitoring data is tagged with high-precision timestamps and temporarily stored in a circular buffer in the form of a structured log stream for near real-time access in subsequent steps.

[0060] As a preferred embodiment, due to the high dimensionality and noise of the original monitoring data, directly inputting it into the model is inefficient. Based on step S200, the most discriminative "key feature set" is extracted, specifically including the following sub-steps:

[0061] S210, Feature Acquisition and Construction: In the initial stage, up to 50 or more low-level indicators can be collected. Based on this, higher-level composite features are constructed using domain knowledge to facilitate the assessment of current system computation-communication overlap and memory status. For example, "Computation-Communication Overlap Rate" = (Time of simultaneous computation and communication) / (Total computation time); "Memory Pressure Index" = Current memory usage / Total memory; "Expected Bubble Duration" = Length of the next idle window predicted based on historical patterns.

[0062] S220. Feature Selection: Using statistical methods (such as Pearson correlation coefficient) or model-based methods (such as feature importance in XGBoost), rigorously select 10-15 key features most relevant to the target variable "iteration delay increment introduced by recomputation" to form an initial key feature set. These key features should comprehensively reflect the coupling relationship between computation, communication, memory, and the bubble.

[0063] S230. Feature Normalization and Encoding: Numerical features in the initial key feature set are normalized using Min-Max or Z-Score, and categorical features (such as the type of operator currently being executed) are One-Hot encoded to ensure the numerical stability of the input vector and the convergence of the model.

[0064] In a preferred embodiment, step S300, which serves as the "brain" of the invention, is divided into two stages: offline training and online reasoning.

[0065] S310, Offline Training Phase:

[0066] S311: Constructing diverse training sets: Running numerous training experiments on various typical large models (such as BERT-base, GPT-2-medium, ViT-Large) and different hardware configurations (such as A100 / V100 clusters). In each experiment, the insertion position of recomputation is systematically perturbed (e.g., before and after each possible micro-batch boundary, each communication synchronization point, etc.) to generate samples covering various scenarios.

[0067] S312: Precise Labeling: For each sample, accurately measure its end-to-end iteration latency and record it along with contextual information such as memory usage and bubble size. Label the recomputation strategy that minimizes the iteration latency of that sample as the "optimal strategy label," thus forming a training feature set with the optimal strategy label.

[0068] S313: Model Training and Validation: Select a supervised learning algorithm suitable for processing structured data (such as random forest, gradient boosting tree GBDT), and train the machine learning decision model using a training feature set with optimal policy labels. Prevent overfitting through techniques such as cross-validation and early stopping, ultimately obtaining a machine learning decision model with strong generalization ability.

[0069] S320, Online Reasoning Stage:

[0070] The key feature set is input into the trained machine learning decision model.

[0071] The output of the machine learning decision model is no longer a single "yes / no" decision, but a list containing multiple candidate time windows and their corresponding prediction delays.

[0072] The comprehensive analysis within the machine learning model is specifically used to simulate the chain reaction after recomputation is inserted into each candidate window, including: Will a surge in temporary activation values ​​lead to a memory overflow? Will it disrupt an efficient computation-communication overlap, thus turning a small bubble into a large one? Only windows that pass the memory safety check and have the lowest predicted net latency will be retained.

[0073] In a preferred embodiment, step S400 specifically includes:

[0074] Evaluation strategy: Select the window with the shortest prediction latency from all windows that meet the memory limit. For example, if the model outputs two candidate windows (communication gaps A and B) with prediction latencies of ΔT_A and ΔT_B respectively, then select the window corresponding to min(ΔT_A, ΔT_B).

[0075] Preferably, the deterministic overhead caused by recalculation within the candidate time window, including memory bottlenecks or expanding existing pipeline bubbles, is used as the metric for predictive latency.

[0076] For example, the time required to recalculate the forward propagation of a Transformer Block can be estimated using historical averages or static analysis. The final decision is based on the formula: "Total prediction latency = Original latency + Recalculation overhead - Utilized bubble duration".

[0077] In a preferred embodiment, step S500 specifically involves: once the optimal timing is selected, the system needs to dynamically modify the execution flow during runtime.

[0078] Runtime code transformation: This invention preferably employs source-to-source or Just-In-Time (JIT) compilation techniques. For example, using compiler frameworks such as LLVM or TVM, a recalculated function call instruction is dynamically injected when the program execution flow is detected to be about to reach the predicted optimal insertion point. This approach is completely transparent to upper-layer applications and requires no modification to user code.

[0079] Alternative: Intervention can also be made at the scheduler level of the training framework, for example, by registering a custom Hook in PyTorch's Autograd Engine to trigger recalculation logic under specific conditions.

[0080] This invention combines advanced machine learning techniques with deep system performance analysis, successfully upgrading the key optimization technique of recomputation from "static and passive" to "dynamic, proactive, and intelligent," providing strong technical support for the efficient and stable training of large models.

[0081] This invention also provides a device for selecting the timing of recomputation during large model training, comprising:

[0082] The status monitoring module is used to monitor the system status parameters in real time during the training process of a large model. The system status parameters include the progress of computation tasks, the progress of communication tasks, the device memory usage, and pipeline bubbles generated by the overlap between computation and communication tasks.

[0083] The feature engineering processing module is used to perform feature engineering processing on the system state parameters and select the key feature set that has the highest correlation with the performance impact of recomputation operations.

[0084] The intelligent decision-making module, based on the key feature set, comprehensively analyzes the spatiotemporal characteristics of the production line bubbles, current and predicted memory limitations, through the machine learning decision model, and outputs candidate time windows for recalculation and corresponding prediction delays; the candidate time window with the shortest prediction delay is selected as the optimal timing.

[0085] The execution module triggers a recalculation operation at the optimal time.

[0086] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0087] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0088] This invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the methods described in any of the embodiments.

[0089] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0090] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0091] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for selecting the timing of recalculation during large model training, characterized in that, Includes the following steps: Real-time monitoring of system status parameters during large model training process, including computation task execution progress, communication task execution progress, device memory usage, and pipeline bubbles generated by the overlapping relationship between computation and communication tasks; The system state parameters are subjected to feature engineering to select the key feature set that has the highest correlation with the performance of recomputation operations; The key feature set is input into a pre-trained machine learning decision model. The machine learning decision model comprehensively analyzes the spatiotemporal characteristics of the pipeline bubble, the current and predicted memory constraints, and outputs candidate time windows for recomputation and the corresponding prediction delay. The comprehensive analysis includes: assessing whether maximizing computational communication overlap will cause memory overflow. If memory overflow occurs, the number of input samples is reduced or recomputation is initiated. The model also predicts the impact of the increased computation time after recomputation on the original computational and communication overlap relationship, and whether new pipeline bubbles will be generated or existing bubbles will be expanded. If new pipeline bubbles are generated, the timing of adding recomputation is reconsidered. Select the candidate time window with the shortest prediction latency as the optimal timing; including: triggering memory bottlenecks caused by recalculation or expanding existing pipeline bubbles within the candidate time window as evaluation indicators for prediction latency. The recalculation operation is triggered at the optimal time.

2. The method according to claim 1, characterized in that, The real-time monitoring of system state parameters during the large model training process includes: integrating a performance profiling agent into the training framework's underlying or driver layer to achieve real-time monitoring of the system state parameters.

3. The method according to claim 1, characterized in that, The feature engineering process includes: Based on the underlying indices of the system state parameters, composite features of different system state parameters are constructed using domain knowledge; From the composite features, the composite features most correlated with the target variable of the iterative delay increment introduced by recalculation are selected as key features to obtain the initial key feature set; The numerical features in the initial key feature set are standardized and scaled, and the categorical features are one-hot encoded to obtain the key feature set.

4. The method according to claim 1, characterized in that, The pre-trained machine learning decision model was obtained through the following offline training process: Run multiple samples that include recomputation operations, configure different recomputation timing selection strategies for the core data objects in each training sample, collect the corresponding system state parameters as training features, and record the training iteration delay under each strategy. Each training sample is labeled with an optimal policy label to form a training feature set with the optimal policy label. The optimal policy label is the recomputation timing selection policy that can achieve the shortest training iteration delay in the configuration of the sample. Based on the training feature set with the optimal policy label, a machine learning decision model is trained using a supervised learning algorithm to obtain a pre-trained machine learning decision model.

5. The method according to claim 1, characterized in that, The step of triggering the recalculation operation at the optimal time includes: inserting the recalculation operation into the code position corresponding to the optimal time through source-to-source or just-in-time compilation technology.

6. The method according to claim 1, characterized in that, The machine learning decision model uses one of the following algorithms: random forest, decision tree, random tree, bagging algorithm, or REPTree.

7. A device for selecting the timing of recalculation during large model training, characterized in that, include: The status monitoring module is used to monitor the system status parameters in real time during the training process of large models. The system status parameters include the execution progress of computing tasks, the execution progress of communication tasks, the device memory usage, and pipeline bubbles generated by the overlap between computing and communication tasks. The feature engineering processing module is used to perform feature engineering processing on the system state parameters and filter out the key feature set that is most correlated with the performance impact of recomputation operations. The intelligent decision-making module, based on the key feature set, comprehensively analyzes the spatiotemporal characteristics of the pipeline bubbles, current and predicted memory limitations, through a machine learning decision model, and outputs candidate time windows for recomputation and corresponding prediction delays. The comprehensive analysis includes: assessing whether maximizing computation-communication overlap will cause memory overflow; if memory overflow occurs, reducing the number of input samples or initiating recomputation; predicting the impact of the increased computation time after recomputation on the original computation-communication overlap relationship, and whether new pipeline bubbles will be generated or existing bubbles will expand; if new pipeline bubbles are generated, reconsidering the timing of recomputation; selecting the candidate time window with the shortest prediction delay as the optimal timing; including using the memory bottleneck caused by recomputation or the expansion of existing pipeline bubbles within the candidate time window as the evaluation metric for prediction delay. The execution module triggers a recalculation operation at the optimal time.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the method as described in any one of claims 1-6.

Images