Large model gradient cutting training optimization system for reducing video memory occupation

Abstract

The invention discloses a large model gradient clipping training optimization system for reducing video memory occupation, and relates to the technical field of artificial intelligence and machine learning, and the system comprises a video memory monitoring and budgeting module which is used for carrying out the real-time collection and statistics of the video memory usage condition in the training process, predicting the video memory peak value of the next training step, and generating a video memory budgeting signal; and the gradient adaptive cutting module is used for carrying out multi-section adaptive cutting on the model parameter gradient according to the video memory budget signal so as to control the gradient amplitude and keep the gradient distribution stable. According to the invention, through the real-time prediction and feedback mechanism of the video memory monitoring budget module, the system can dynamically sense the pressure of the video memory and intelligently adjust the strategy, thereby avoiding the defects of static configuration; the gradient adaptive cutting module is combined with technologies such as semantic section cutting, step stable window and noise consensus gating to realize refined and adaptive control of the gradient, gradient explosion is effectively prevented, gradient information is maintained to the greatest extent, and convergence stability is ensured.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and in particular to a large model gradient pruning training optimization system that reduces GPU memory usage. Background Technology

[0002] In the context of the rapid development of artificial intelligence technology, the training of large models (such as Transformer models, diffusion models and multimodal models) has become mainstream. These models usually have billions or even trillions of parameters, and their training process places extremely high demands on computing resources, especially GPU memory. Traditional training methods and optimization strategies often face problems such as GPU memory overflow, low training efficiency and unstable convergence when faced with such large models.

[0003] Existing technologies are often isolated and static optimization measures, making it difficult to flexibly adapt to the dynamic changes in memory usage during large model training. While gradient pruning can prevent gradient explosion, blind pruning may distort the gradient distribution and affect model convergence. The adjustment of batch strategies lacks a comprehensive consideration of memory, computation, and convergence effects. Existing systems lack an integrated solution that can coordinate and adaptively optimize multiple aspects such as memory management, gradient optimization, batch scheduling, and numerical stability. As a result, developers still need to invest a lot of effort in manual tuning during large model training, and it is difficult to achieve optimal training results. Especially when further expanding the model size or increasing the equivalent batch size under a fixed memory configuration, existing technologies are often inadequate. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a large model gradient pruning training optimization system that reduces GPU memory usage.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a large model gradient pruning training optimization system for reducing GPU memory usage, comprising the following six functional modules: a GPU memory monitoring and budget module for real-time collection and statistics of GPU memory usage during training, predicting the peak GPU memory usage in the next training step and generating a GPU memory budget signal; a gradient adaptive pruning module for performing multi-segment adaptive pruning of model parameter gradients based on the GPU memory budget signal to control gradient amplitude and maintain gradient distribution stability; and a batch accumulation scheduling module for dynamically adjusting the micro-batch size, gradient accumulation steps, and sequence length upper limit under GPU memory budget constraints, thereby achieving a balance between GPU memory and computational throughput. The system is structured as follows: a balance module; a recalculation and sharding optimization module; a loss scaling protection module; a training convergence monitoring module; a training convergence monitoring module; and a training convergence control module. The training convergence monitoring module is used to perform causal analysis and priority adjustment on the running status of each module, calculate convergence scores, generate policy feedback, and establish a system self-learning closed loop. The six modules communicate with each other via memory budget signals, training feedback signals, and policy instructions, thereby achieving a collaborative closed loop of memory usage control, gradient stabilization, and training efficiency optimization.

[0006] As a further description of the above technical solution: The memory monitoring and budgeting module includes: memory sampling and hierarchical statistical technology for collecting memory usage, remaining capacity, and operator-level percentage of each computing device through the memory interface, and dividing the memory information into model parameter memory layer, optimizer state layer, activation cache layer, and communication buffer layer; memory trend prediction technology for predicting the next memory peak based on historical memory curves and batch configuration; risk band constraint technology for dividing the memory state into safe zone, tense zone, and over-limit zone based on the prediction results, and sending adjustment signals to subsequent modules when entering the tense zone; and memory feedback and correction technology for calculating prediction errors and adjusting prediction parameters to achieve self-learning and correction of the memory budget.

[0007] As a further description of the above technical solution: The gradient adaptive pruning module includes: a semantic segment pruning technique for dividing parameters into different semantic segments based on the model structure and setting independent pruning rules for each segment; a step stabilization window technique for calculating the gradient mean, variance, and peak frequency within a fixed-length window, generating a steady-state score, and adjusting the pruning intensity; a noise consensus gating technique for synchronously judging abnormal gradients across multiple computing nodes, triggering global suppression only when a majority of nodes consistently detect an anomaly; and a topological affinity suppression technique for identifying strongly correlated parameter groups in the computation graph and maintaining their gradient ratios to prevent local oscillations.

[0008] As a further description of the above technical solution: The batch cumulative scheduling module includes: a three-domain coupled scheduling technique that uses the micro-batch size, gradient accumulation steps, and sequence length as joint scheduling parameters to perform synchronous adjustment based on the memory status; a self-game exploration technique that constructs a pool of candidate parameter combinations, rotates experiments in the short term, and retains the optimal combination based on the payoff and risk scores; a cross-node heterogeneous shaping technique that allocates the load shape based on the memory capacity and bandwidth capabilities of each device in a multi-computing environment to achieve heterogeneous synchronization; and a window order preservation and cooling-off technique that sets a policy cooling-off period and a freeze period to limit frequent switching and ensure the stability of the training process.

[0009] As a further description of the above technical solution: The recomputation and fragmentation optimization module includes: fine-grained critical path recomputation technology to identify subgraphs with high memory usage and low gradient sensitivity, and perform activation recomputation only on these subgraphs to reduce memory usage; three-layer fragmentation peak-shifting technology to divide parameters and optimizer states into layer fragmentation, tensor block fragmentation, and time slice fragmentation, and load and unload them in different time windows; compression, communication, and computation pipelining technology to perform parameter compression, communication transmission, and reverse computation in parallel during the computation process to achieve resource peak-shifting; and sample-level recomputation and lightweight caching technology to perform local recomputation on samples with high memory usage, while other samples retain lightweight caching to reduce peak memory usage.

[0010] As a further description of the above technical solution: The loss scaling protection module includes: multi-granularity early warning mapping technology for deploying probes at the operator, hierarchical, and global levels to generate stability heatmaps and detect numerical anomalies; hierarchical rollback and progressive recovery technology for performing local scaling rollback when anomalies occur in a local layer, performing global rollback when necessary, and restoring the scaling coefficient in small steps during the recovery phase; strategy linkage guardrail technology for establishing linkage logic with the gradient adaptive pruning module and the batch accumulation scheduling module to control the parameter adjustment order during scaling contraction and recovery; and drift adaptive memory technology for recording abnormal numerical characteristics and loading a conservative scaling template when a similar distribution is detected.

[0011] As a further description of the above technical solution: The training convergence monitoring module includes: causal trajectory modeling technology for recording the correlation events between memory changes, policy actions, and training results; policy priority self-evolution technology for dynamically adjusting the policy priority and amplitude range of each module based on causal benefit statistics; interpretable convergence scoring technology for calculating the convergence score based on convergence speed, stability, effective gradient ratio, and generalization index, and outputting the reasons for score changes; and canary release and safe rollback technology for partially enabling and monitoring the convergence score when a new policy is introduced, and rolling back to the previous stable configuration if the score drops below a preset threshold.

[0012] As a further description of the above technical solution: The memory budget signal output by the memory monitoring budget module is jointly used by the gradient adaptive pruning module, the batch cumulative scheduling module, and the recomputation fragmentation optimization module to form a unified basis for memory constraints; the training convergence monitoring module dynamically adjusts the global strategy based on the feedback information from these modules.

[0013] As a further description of the above technical solution: The system's training process includes the following steps: the memory monitoring budget module generates a memory budget signal; the gradient adaptive pruning module performs hierarchical segment pruning based on the budget signal; the batch accumulation scheduling module adjusts the micro-batch size, accumulation steps, and sequence length based on the memory status; the recalculation and sharding optimization module performs recalculation and peak-shifting operations according to the critical path and sharding strategy; the loss scaling protection module detects numerical anomalies and adjusts scaling parameters; and the training convergence monitoring module merges data and adjusts the priority of each module's strategy to form a closed-loop optimization.

[0014] As a further description of the above technical solution: The system is suitable for distributed training scenarios of Transformer-type models, diffusion models, and large multimodal models. Under the hardware condition of fixed memory, it can realize the expansion of model parameter scale or equivalent batch improvement, while maintaining convergence speed and stability without decreasing.

[0015] The present invention has the following beneficial effects: 1. In this invention, firstly, through the real-time prediction and feedback mechanism of the memory monitoring budget module, the system can dynamically sense memory pressure and intelligently adjust strategies to avoid the shortcomings of static configuration; the gradient adaptive pruning module combines semantic segment pruning, step-stable window, and noise consensus gating technologies to achieve fine-grained and adaptive control of gradients, effectively preventing gradient explosion while maintaining gradient information to the greatest extent and ensuring convergence stability; the batch accumulation scheduling module performs three-domain coupled scheduling of micro-batch, accumulation steps, and sequence length, and under the constraint of memory budget, utilizes self-game exploration and cross-node heterogeneous shaping to achieve a dynamic balance between memory and computational throughput, greatly improving training efficiency and resource utilization; the recomputation and fragmentation optimization module optimizes the peak memory usage from global to local through fine-grained critical path recomputation and three-layer fragmentation peak loading, and combines compression, communication, and computational pipeline optimization to significantly reduce memory requirements without significantly increasing computational overhead.

[0016] 2. In this invention, the loss scaling protection module provides robust numerical stability assurance for mixed-precision training through multi-granularity early warning maps, hierarchical backoff and progressive recovery, as well as strategy linkage with other modules; the training convergence monitoring module, as the "brain" of the system, realizes the self-learning and adaptive closed loop of the entire system through causal trajectory modeling, policy priority self-evolution, interpretable convergence scoring and canary release mechanism, enabling the system to perform global optimization according to the actual situation in the training process. Ultimately, under the hardware condition of fixed memory, it can expand the model parameter scale or increase the equivalent batch size, while maintaining convergence speed and stability without decreasing, significantly reducing the threshold and cost of training advanced artificial intelligence models. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall system architecture of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Reference Figure 1 This invention provides an embodiment of a large model gradient pruning training optimization system that reduces GPU memory usage, comprising the following six functional modules: a GPU memory monitoring and budget module for real-time acquisition and statistics of GPU memory usage during training, predicting the peak GPU memory usage in the next training step and generating a GPU memory budget signal; a gradient adaptive pruning module for performing multi-segment adaptive pruning of model parameter gradients based on the GPU memory budget signal to control gradient amplitude and maintain stable gradient distribution; a batch accumulation scheduling module for dynamically adjusting the micro-batch size, gradient accumulation steps, and sequence length upper limit under GPU memory budget constraints to achieve a balance between GPU memory and computational throughput; and a recalculation module. The computational sharding optimization module is used to recalculate the critical path based on the computational graph structure, and to shard and stagger the loading of parameters and optimizer states to reduce memory peaks; the loss scaling protection module is used to dynamically adjust the loss scaling coefficient during mixed-precision training to prevent gradient overflow and maintain numerical stability; the training convergence monitoring module is used to perform causal analysis and priority adjustment on the running status of each module, calculate the convergence score and form policy feedback to establish a system self-learning closed loop; the six modules communicate with each other through memory budget signals, training feedback signals and policy instructions to achieve a collaborative closed loop of memory usage control, gradient stabilization and training efficiency optimization.

[0020] The memory monitoring and budget module is the starting point and global constraint benchmark of the system's operation process, undertaking four functions: memory status awareness, trend prediction, budget signal generation, and feedback correction. It runs continuously after training begins, collecting dynamic data on GPU memory usage in real time, establishing a hierarchical statistical model, and generating memory risk prediction results based on time-series changes. The output "memory budget signal" is used by the subsequent five modules as the basis for strategies such as gradient pruning intensity, micro-batch scheduling range, recomputation trigger ratio, and numerical scaling upper limit, thus forming a memory-aware driven system adaptive adjustment closed loop. This forms a budget-driven dynamic training control mechanism, where each module uses the memory budget as its behavioral boundary, ensuring precise allocation of memory resources in both time and space dimensions, avoiding memory waste or overflow. Memory sampling and hierarchical statistical techniques: The system calls the memory management API through the driver layer interface to collect real-time memory usage, free capacity, memory allocation table, and operator-level distribution data for each GPU device at millisecond intervals. This process is executed continuously by an independent thread to ensure no interference with the main training computation. The sampled memory information is standardized and divided into four memory layers: model parameter memory layer, optimizer state layer, activation cache layer, and communication buffer layer. Each layer has an independent statistical cache to record its instantaneous usage, average usage, and growth rate. After each training step, the system performs inter-layer aggregation to create a complete snapshot of the memory distribution. This hierarchical structure allows the system to pinpoint the source of memory anomalies. When the growth rate of the activation cache layer exceeds the average level, it indicates that gradient backpropagation paths may not be released in time, triggering local garbage collection and cache refresh instructions. Through this hierarchical mapping, the memory monitoring module can achieve fine-grained control over each type of memory resource. , :time Time of the first The smooth growth rate of each memory layer (model parameter layer, optimizer state layer, activation cache layer, communication buffer layer), The growth rate of the same layer at the previous moment. :time The sampled first Real-time memory usage of the layer The memory usage of this layer at the previous moment. : The time interval (seconds) between two samplings. : An exponential smoothing coefficient used to suppress transient noise. Used to calculate the dynamic growth rate of memory in each layer to detect memory spikes in abnormal layers, providing a stratification basis for subsequent risk assessment. Memory Trend Prediction Technology: To achieve forward-looking prediction of memory usage, a temporal modeling mechanism is introduced. Its inputs are the total memory sequence of the most recent training steps, the changes in the proportion of each layer, the current batch configuration, and the moving average of the gradient norm. A lightweight temporal prediction algorithm (which can be a simplified variant of linear regression or Long Short-Term Memory (LSTM) networks) is used to estimate the trend of memory usage changes in the next one to two steps. The prediction output is the upper and lower limits of the memory peak in the next training step and its confidence score. Three prediction modes are divided according to the confidence level: high confidence mode directly outputs the budget signal; medium confidence mode outputs the signal after superimposing a safety factor; low confidence mode enters a conservative prediction mode to prevent misjudgment leading to memory overflow. The parameters of the prediction model are continuously updated during training to ensure long-term adaptability. Through trend prediction, the system can identify memory peak risks in advance. During the large-scale data loading or gradient burst phase, it can issue a stress signal in advance, enabling downstream modules to take measures such as reducing batch size or activating recalculation in advance, thus achieving proactive prevention and control of memory risks. , : Predicted peak memory usage for the next training step The bias term of the model. : Past The total observed value of the video memory in the step. : Weighting coefficients of autoregressive time series data. Each memory layer in the past The total magnitude of the step percentage change vector. Weighting coefficients for changes in layer proportions. : The moving mean of the gradient norm within the current window, The weight of the gradient norm on changes in video memory. : Batch configuration parameters for the current training step (micro-batch size, etc.) The ratio of batch size to memory demand is calculated using a linear regression or a simplified LSTM layer expansion, outputting the next step's peak memory estimate. Memory risk zone constraint technique: Based on memory prediction results, memory usage is divided into three risk zone intervals: safe zone, tense zone, and over-limit zone. When memory usage is below 80% of the budget, it is in the safe zone, and the system allows all modules to operate normally. When usage reaches 80% to 95%, it enters the tense zone, issuing a "memory tense signal" to prompt downstream modules to implement mild suppression strategies, such as reducing micro-batch size or increasing gradient pruning intensity. When usage exceeds 95% of the budget, the system enters the over-limit zone, triggering a forced rollback mechanism. The forced rollback mechanism includes two steps: first, freezing the update step size of the batch scheduling module to prevent further expansion of memory usage; second, initiating the real-time compression process of the recalculation and sharding module to perform local recalculation and parameter unloading operations on high-usage layers, thereby reducing peak memory usage without terminating training. This risk zone division mechanism allows the system to self-adjust before memory usage approaches a critical value, achieving continuous stability during training. , :time Risk zone coefficient, range of values This provides risk signals to downstream modules. :time Actual monitored video memory usage, : The maximum available video memory budget for the current device or parallel group, when Indicates the safe zone; Indicates a tense area; This indicates an over-limit zone and triggers a forced rollback mechanism. Memory feedback and correction learning techniques: After each training cycle, the predicted peak value is compared with the actual memory peak value to calculate the error deviation. The error is written into the self-learning unit to adjust the parameter weights of the prediction model. If the system repeatedly shows deviations in the same direction (e.g., continuously underestimating the peak value), the prediction model automatically increases safety redundancy; if the error oscillates bidirectionally, the learning rate is reduced to smooth the adjustment. This self-learning process continues throughout the entire training lifecycle, gradually bringing the memory prediction capability closer to the actual distribution. A memory behavior profile is retained, recording the memory usage characteristics at different model stages. When the system detects that the memory curve of the current training stage is highly similar to the historical profile, it can directly call the budget template corresponding to that stage without re-predicting, thereby improving response speed and prediction accuracy. The introduction of correction learning enables the memory monitoring budget module to have continuous self-evolution capabilities. , : Step memory Prediction Model No. The current value of each parameter. The parameter values ​​after correction and update. Learning rate controls the magnitude of error correction. :step True peak memory, :step Predicted peak memory usage, Predictive model parameters gradient sensitivity, Redundancy adjustment coefficient: Increases safety redundancy when continuous deviations are in the same direction. The sign function controls the redundancy adjustment direction based on the direction of continuous deviation. Online correction is achieved by comparing the predicted and actual differences, enabling the memory prediction model to have self-learning and self-evolution capabilities. The memory monitoring and budgeting module runs as a background daemon after system startup. Before each training step, it updates the memory status and outputs the latest budget signal; after each training step, the module collects actual memory usage data and feeds it back to the prediction model. This process forms a complete closed loop of "sampling—prediction—constraint—feedback". When the memory budget signal is referenced by downstream modules, the system automatically adds a risk level label. For example, the gradient adaptive pruning module can adjust the pruning threshold based on the label; the batch cumulative scheduling module changes the micro-batch size based on the label; and the recalculation fragment optimization module determines whether to enable local recalculation based on the label. Through this signal transmission method, the memory budgeting module becomes the control center and coordination hub of the entire system. In long-term training tasks, the memory monitoring and budgeting module can effectively prevent overflow or abnormal termination caused by sudden increases in memory. By combining hierarchical sampling and trend prediction, the system achieves real-time visibility and predictive control of memory without significantly increasing computational overhead. Its feedback correction mechanism ensures that the prediction model maintains high accuracy at different stages (such as the warm-up period, stabilization period, and convergence period). The introduction of the memory risk zone mechanism allows the modules to work collaboratively even when the memory is close to its limit, avoiding resource waste caused by excessive conservatism. At the system-wide level, the operating results provide a dynamic, continuous, and reliable basis for memory constraints for subsequent modules, thereby realizing the transformation from passive protection to active budgeting and building the foundational support for the entire memory optimization system.

[0021] The gradient adaptive pruning module dynamically adjusts the gradient pruning threshold for each parameter segment based on the memory budget signal output by the memory monitoring and budgeting module. This prevents memory spikes or training oscillations caused by abnormal gradient amplification under memory-constrained conditions. This process runs throughout the entire training phase, and its control results directly affect numerical stability, gradient distribution balance, and model convergence speed. It not only constrains the gradient magnitude but also performs layered and dynamic adjustments to the pruning strategy based on model structure and historical stability. Through collaborative pruning of different semantic segments, different time windows, and multiple GPU nodes, the gradient control process achieves both global coordination and local adaptability. Semantic segment pruning technology is used to differentiate gradient control at the model structure level. Before training begins, the system reads the model's structural information, including layer names, weight tensor shapes, and parameter category labels. Based on this information, the model parameters are divided into multiple semantic segments, distinguishing attention weight segments, feedforward network weight segments, and normalized bias segments in the Transformer model. Each segment establishes an independent gradient statistics cache to record the gradient mean, variance, and extreme value proportions within a specified time window. After each backpropagation, the system calculates the pruning intensity of that segment based on the memory budget signal and the gradient distribution state of each segment. For highly sensitive segments, such as normalization layers or bias parameter segments, a more lenient pruning strategy is adopted to ensure the continuity of parameter updates; while for less sensitive segments, such as feedforward layer weights or projection matrices, stronger gradient limiting can be implemented to control memory usage. The adjustment ratio is calculated independently for each segment during pruning to avoid over-pruning or loss of gradient information due to a uniform threshold. The core of semantic segment pruning is to construct a segment index through parameter labels and a gradient mapping table, enabling the system to quickly locate the parameter positions of each segment and execute the corresponding pruning rules. This index is dynamically maintained during training and automatically updated when the model structure changes or the memory budget is adjusted. Segment gradient extraction: , In training steps Belongs to semantic segment The gradient column vector, Semantic segment A binary selection matrix (selecting the dimension corresponding to the parameters of the segment). Training steps The full model gradient column vector, Semantic segment index (e.g., attention weights, feedforward weights, normalized biases, etc.). Segment gradient norm: , Training steps semantic segment The 2-norm, Training steps semantic segment No. Gradient values ​​in each dimension, summation index : Parameter dimension index within the segment. Baseline clipping threshold: , Training steps semantic segment The baseline clipping threshold, Semantic segment The static reference threshold, Semantic segment Sensitivity score (the higher the score, the more gentle the cropping is required). Sensitivity scaling factor (magnification) (the impact) Training steps The memory risk is tagged (safe / critical / overlimit). Training steps The memory stress index (the ratio of the next standardized peak value to the upper limit of the budget). Training steps Budget forecast confidence patterns (high / medium / low). Threshold scaling function based on risk zone and pressure. : Threshold scaling function based on confidence pattern. Final pruning threshold: , Training steps semantic segment The final cropping threshold, Training steps semantic segment steady-state fraction, Steady-state target level (above this value tends to be relaxed). The linear adjustment coefficient of the steady-state fraction with respect to the threshold. Semantic segment The lower / upper bound of the threshold, : scalar Clip to range Segment trimming operator: , Training steps semantic segment The clipped gradient vector, Numerical stability constant to prevent the denominator from being zero. The step stability window technique is used to determine the stability of gradient changes over time.

[0022] The system establishes a fixed-length time window for each semantic segment, covering the most recent 32 training steps. After each training step, the module writes the mean, variance, and peak frequency of the gradient for that segment into the window cache. By statistically analyzing the gradient features within the window, a steady-state score is calculated for that segment to measure the smoothness of gradient changes. A higher steady-state score indicates smaller gradient fluctuations in that segment, allowing for a more relaxed clipping threshold in the next training step; a lower steady-state score indicates instability in that segment, automatically increasing the clipping intensity. Through this mechanism, the gradient clipping process is no longer a single-step reaction but an adjustment based on trends over a period of time, thus avoiding frequent threshold oscillations and policy switching. To ensure statistical stability, outliers in the window are replaced with a median-limited method, ensuring that individual extreme gradients do not have an excessive impact on the steady-state evaluation. This mechanism allows the system to gradually adjust the gradient clipping intensity smoothly, improving overall training stability. In a multi-GPU distributed training environment, gradient noise inconsistencies may occur between computing nodes due to local data differences or communication delays. Robust clipping: , , Window position (Corresponding steps) The statistical value after the limit is ) Window position The original statistical values ​​(available segment norms) ), Window length, : Median of the window MAD robustness scale : Limiting factor; the larger the value, the more lenient the limit. Steady-state fraction: , , , , : Window mean after limiting Standard deviation of the window after limiting. Peak frequency (proportion exceeding threshold) Peak threshold coefficient Steady-state fraction: the larger the value, the more stable the state. Weights for coefficient of variation penalty and peak penalty. Numerical stability constant : Indicator function (condition 1 is satisfied).

[0023] Noise consensus gating technology is used to establish a consensus mechanism for gradient anomaly detection in a distributed environment. After performing local backpropagation, each GPU node first performs a lightweight pruning based on the current threshold and generates a corresponding noise label indicating whether there are abnormal gradient spikes in that segment. The label generation logic is based on local statistics of the gradient distribution and the memory pressure index. Nodes synchronize through a communication interface and vote on the noise labels. Only when more than half of the nodes detect anomalies in the same segment is the system determined to be a globally anomalous segment and triggers a forced suppression operation. If only a few nodes detect anomalies, it is considered a local fluctuation and global suppression is not performed. Through this majority consensus mechanism, the system can effectively filter out erroneous pruning caused by misjudgment by a single GPU or communication jitter, ensuring the consistency and robustness of distributed gradient updates. The gradient adaptive pruning module records the results of each consensus for subsequent statistical learning. When certain segments frequently trigger consensus suppression in multiple training stages, the system marks them as high-risk segments and prioritizes them in future training, thereby improving the predictability and stability of the overall training. Node local anomaly labeling: , , , , :node For section Standardized outlier scores, :node Section The gradient norm, :node The moving mean and moving standard deviation, : Exponential sliding coefficient Numerical stability constant :node The local area abnormal label, Abnormal z-score threshold This step section The final cropping threshold, Memory stress index. Pressure amplification factor (the higher the pressure, the easier it is to cause abnormalities), "Logical OR. Majority vote:" Threshold suppression: , The percentage of nodes judged as abnormal. Whether to trigger the global suppression gating, : Number of participating GPU nodes Voting threshold (e.g., majority agreement). Consensus inhibition strength coefficient, ”: In-place update assignment operator.

[0024] Topological affinity suppression (TAS) maintains stable gradient ratios among parameters and prevents local gradient oscillations from propagating to other layers by analyzing the structural dependencies in the computation graph. When constructing the model's computation graph, the system analyzes the topological connections between parameters, identifying parameter pairs or groups with strong dependencies on the computation path, such as query weights and key weights in the attention mechanism, and up and down projection matrices in the feedforward network. During training, when the system detects significant differences in the gradient direction or magnitude of these related parameter groups, it automatically performs coupling constraint operations, i.e., maintaining stable gradient relationships through scaling. This avoids gradient direction deviations caused by over-pruning or widening of certain local layers, thus maintaining the optimization consistency of the entire network. The core objective of TAS is to prevent local gradient imbalances from escalating into global convergence oscillations. The system employs a combination of static structure and dynamic statistics when performing topological analysis. The static part comes from the graph structure defined in the model definition, while the dynamic part is provided by the gradient covariance matrix collected during training. When dynamic statistics show a significant increase in the correlation of a certain set of parameters, the affinity constraint strength for that set is automatically increased; conversely, it is gradually weakened. This dynamic adjustment maintains a reasonable gradient coupling relationship at different training stages, enabling an evolution from static topological constraints to adaptive equilibrium.

[0025] Within-group correlation estimates: , , , Related Groups The moving average vector (e.g., composed of two gradient norms). : The current statistical vector of the relevant group (e.g. ), Related Groups The recursive covariance matrix, Smoothing coefficient Related Groups The correlation coefficient, Numerical stability constant. Amplitude: Consistency in direction: , : parameter subvector and gradient magnitude ratio, : parameter subvector and Direction cosine, : parameter subvector , The gradient column vector, ": Vector dot product, Numerical stability constant. Affinity strength: Coupling constraints: , , , Related Groups The affinity strength (the higher the correlation, the greater the affinity). Sigmoid function Affinity strength gain coefficient Relevant threshold, Related Groups Target amplitude ratio, : Amplitude ratio to corrected power exponent, Direction-aligned blended weights Direction cosine trigger threshold : exist The projection on : Amplitude ratio compared to the corrected gradient : The gradient after orientation alignment ( Similarly), Numerical stability constant. The gradient adaptive pruning module maintains real-time synchronization with the memory monitoring and budgeting module during system operation. When the memory budget signal indicates that the system has entered a stressful region, the pruning threshold and stability window weight of each semantic segment are immediately adjusted according to the budget level. For high-risk segments, the system will prioritize tightening the pruning intensity and limiting the fluctuation range within the window; once the memory returns to a safe region, the restrictions are gradually relaxed according to the steady-state score. The gradient adaptive pruning module is linked with the loss scaling protection module and the batch cumulative scheduling module.

[0026] Risk-stress scaling function: , Risk-stress scaling factor Memory risk labels (safe / critical / overlimit) Memory stress index. : Linear deduction slope under stress and overlimit (usually) ), : Scale the lower limit to avoid the threshold being too small.

[0027] Confidence mode scaling function: , Confidence scaling factor Budget forecast confidence level (high / medium / low). Fixed deduction margin under medium / low confidence levels. Loss scaling linkage: , Current step segment The final pruning threshold (is updated in-place). Training steps loss scaling factor, Training steps loss scaling factor, The coupling exponent of loss scaling with respect to the threshold. Numerical stability constant. Gradient accumulation step linkage: , Current step segment The final pruning threshold (is updated in-place). Training steps The number of gradient accumulation steps, Training steps The gradient accumulation steps, √(1 / 2) factor, approximately maintain the effective gradient scale of a single update as the accumulation steps change. When the loss scaling shrinks, the gradient pruning module temporarily increases the pruning magnitude to prevent overflow; when scaling recovers, it synchronously decreases the pruning strength to maintain continuous learning rate. Through cooperation with other modules, a stable learning mechanism under memory constraints is formed. In long-term training, the adaptive gradient pruning module effectively smooths the gradient distribution through semantic segmentation and step stabilization mechanisms, reducing the impact of instantaneous peaks on memory; noise consensus gating ensures gradient consistency under multi-GPU collaboration; and topological affinity suppression maintains the structural balance between parameters. Its combined effect enables the model to maintain efficient learning capability and stable convergence performance even under memory-constrained conditions. It realizes the transformation from static pruning to dynamic adaptive pruning, evolving gradient control from single-layer limiting to a globally coordinated, cross-layer balanced system, providing stable, intelligent, and scalable core support for the entire memory optimization system.

[0028] The batch cumulative scheduling module dynamically adjusts the micro-batch size, cumulative steps, and sequence length upper limit based on the budget signal from the memory monitoring budget module. This maintains overall throughput and training stability under memory-constrained conditions. Its design goal is to maximize the utilization of computing resources through intelligent scheduling without exceeding the memory budget, while avoiding numerical oscillations caused by excessively large batches or frequent switching. It runs continuously during training and dynamically updates batch parameters at each training stage. Its output not only determines the peak memory usage per step but also affects the frequency of gradient accumulation and optimizer updates, thus serving as a crucial bridge connecting the memory layer and the optimization layer in memory adaptive control. The three-domain coupled scheduling technique is the core algorithm of the batch cumulative scheduling module, used to achieve coordinated control between "micro-batch size, cumulative steps, and sequence length" under memory-constrained conditions. The system treats these three parameters as interrelated dynamic decision variables, corresponding to memory usage, gradient update frequency, and sequence processing load, respectively. Whenever the memory monitoring budget module outputs a new budget interval signal, the batch accumulation scheduling module reads the current memory load status, the throughput performance of the previous stage, and the gradient stability index, and recalculates the resource allocation ratio among these three factors. When the memory is in the safe zone, the system prioritizes increasing the micro-batch size to improve the parallelism of single-step computation; when the memory enters the tense zone, the system simultaneously reduces the micro-batch size and increases the number of accumulation steps to maintain effective gradient statistics, while limiting the upper limit of the sequence length based on the memory change rate, thereby reducing activation cache usage. When the risk of exceeding the limit occurs, the micro-batch size is immediately locked and the sequence length is temporarily shortened to prioritize ensuring the continuity of training. The execution process of the three-domain coupled scheduling is implemented by an independent scheduling loop thread, which re-evaluates the resource configuration at the end of each training cycle, enabling the adjustment to have latency compensation and anti-oscillation characteristics. Internally, the change trajectory of each parameter is retained for adaptive initialization in subsequent training stages, thus enabling the system to have stage memory capabilities. Budget-constraint three-domain joint scoring: , In training steps For candidate triples The overall score (the higher the better). : The micro-batch size of the candidates (number of samples per small step per card / node). : Number of candidate gradient accumulation steps (how many small steps to merge an optimizer update). The upper limit of candidate sequence length (affects activation overhead and operator parallelism). In step Differentiable / segmented differentiable proxies for throughput :step The throughput normalization scale (maximum observed throughput within a historical window). In step Proxying peak video memory, :step Safety margin, :step The budget limit (the available limit after conversion based on device / cluster configuration). In step Stability measure Four weighting coefficients are used to balance returns, risks, stability, and switching costs. The triplet used in the previous step Numerical stability constant to avoid zero denominator. Three surrogate models: , , , :step Computation latency proxy (kernel function time / computing load estimation). :step Synchronous communication cost proxy, :step Other overhead (data preparation, scheduling management). :step The memory proxy coefficients (obtained through online linear / piecewise linear regression). :step Gradient variance estimation (obtained through window statistics). :step The sequence normalization scale, :step The stability surrogate coefficient and sequence exponent.

[0029] Risk zone → Safety margin mapping: , Safety margin: The smaller the margin, the more conservative the proportion of available budget. :step The stress index (standardized intensity of the next peak / upper limit). :step Pressure sensitivity (self-calibrated with long-term error). :step Risk labels (safe / nervous / exceeding limits). Fixed reduction amount under stress / exceeding limits.

[0030] Game-theoretic exploration techniques are used to automatically select the optimal combination from multiple batch configuration schemes, avoiding efficiency degradation caused by manual settings and static parameters. During system initialization, a candidate configuration pool is built, storing multiple possible combinations of micro-batches, cumulative steps, and sequence lengths. Each configuration includes predicted memory usage, expected throughput, and gradient stability metrics. During training, several schemes are periodically selected from the configuration pool for short-term testing, typically within a few training steps. During testing, the system records peak memory usage, throughput, and loss reduction rate in real time, and calculates a comprehensive payoff-risk score. The scoring function is weighted based on the degree to which memory usage approaches the budget limit, the throughput improvement ratio, and gradient stability. After testing, the configuration with the highest score is selected as the official scheduling scheme for the current stage, and low-scoring schemes are marked as eliminated. Eliminated schemes can be re-evaluated in subsequent stages to prevent the loss of potential optimal solutions after changes in model structure or data distribution. The entire self-game process is similar to configuration evolution; the system achieves adaptive search of the parameter space through continuous competition and elimination, finding the optimal strategy that balances memory safety and computational efficiency without external intervention. The results of the self-game exploration also influence the prediction model of the memory monitoring and budgeting module. When the system repeatedly selects the same configuration and achieves stable returns, the memory budgeting module incorporates the memory usage pattern of that configuration into the historical learning template, making memory prediction more accurate, thereby achieving mutual reinforcement between modules. Upper confidence score with risk penalty and uncertainty: , :step The self-game score of the candidate configuration (the higher the score, the better). Candidate Average throughput of the short test window Candidate Average loss rate of decrease (per second / token) during the short test window. Candidate Average peak memory usage during short test windows :step The normalized scale (maximum window value). :step Effective budget, The weights of the three factors: return, convergence, and existing risk. :step The exploration coefficient (the score of insufficiently tested schemes is increased). :step The sum of all trials up to this point Candidate Number of trials completed Candidate Throughput variance (stability uncertainty) of short test window. :step The variance normalization scale, :step The penalty coefficient for unstable schemes.

[0031] Cross-node heterogeneous shaping technology addresses the load imbalance caused by inconsistent memory capacity and communication bandwidth in multi-GPU or multi-node environments. At the start of training, a hardware capability assessment is performed on all nodes, recording each node's memory capacity, communication latency, and bandwidth parameters, and generating a node capability description table. The system calculates the "load shape" for each node, i.e., the matching ratio of micro-batch size to sequence length. Nodes with high memory are allocated larger micro-batches and longer sequences, while nodes with low memory use smaller micro-batches and shorter sequences, maintaining a consistent global step size through distributed gradient accumulation. This ensures that all nodes complete synchronous updates in the same training step, thus avoiding step size misalignment and memory slowdown caused by node heterogeneity. During training, the memory utilization and communication latency of each node are continuously monitored. When a node's memory utilization is significantly lower than average, the micro-batch limit for that node is slightly increased to improve overall load balancing; when a node approaches its memory limit, the data load on that node is automatically reduced, and tasks are redistributed among other nodes. Cross-node heterogeneous shaping technology ensures the coordinated operation of the system in multiple hardware environments, making the distribution of video memory usage tend to be balanced, while maintaining the synchronization and efficiency of global training.

[0032] Node capability factor: Quota allocation: , : No. The ability factor of each node (the larger the factor, the stronger the ability). : No. Effective video memory capacity (available bytes) of each node. : Reference scale for video memory capacity : No. The computing power metrics of each node Computing power reference scale : No. Communication latency of each node (average latency of synchronization / All-Reduce). : No. The computational latency of each node (average time per unit of workload). Weighting index of the three abilities : No. Micro-batch allocation of nodes, : Total number of global target micro-batches (total parallelism expected within budget). Number of nodes Rounding to the nearest integer. Global step size consistency: , sequence adaptation: , A globally consistent gradient accumulation step count (ensuring synchronous updates). The total effective batch size (policy setting) corresponding to each optimizer update. : No. The upper limit of the sequence length of each node. Global sequence target (expected upper limit in the safe zone). To prevent sudden data loss, a video memory buffer is reserved. Unit micro-batch / fixed two types of video memory proxy coefficient (online estimation).

[0033] Utilization-driven lightweight rebalancing: , : No. Micro-batch fine-tuning amount per node : No. The utilization rate of the memory or computing power of each node (0–1). Average utilization rate of all nodes : Rebalancing step size coefficient, : Rebalancing threshold (no action is taken if the deviation is less than the threshold). : Sign function (positive / negative indicates increase / decrease).

[0034] The window-based order preservation and cooling-off mechanism prevents training instability or numerical oscillations caused by frequent switching of batch scheduling strategies. This mechanism defines two key phases: a strategy cooling-off period and a strategy freeze period. During the strategy cooling-off period, continuous modifications to the batch configuration are prohibited; that is, only one scheduling adjustment is allowed within a cooling-off period. The length of the cooling-off period is dynamically determined based on the rate of change of memory and the convergence speed. When the system detects large memory fluctuations or short-term oscillations in the loss curve, it automatically extends the cooling-off period to avoid rapid strategy switching. The strategy freeze period is used to maintain training consistency during the parameter stabilization phase. During the freeze period, key hyperparameters, including the learning rate and regularization coefficient, are locked to prevent gradient statistical bias caused by changes in batch size. The freeze period typically covers a complete accumulation period to ensure the integrity of gradient accumulation statistics. At the end of each cooling-off period, the memory state, training stability, and throughput performance are reassessed. If the conditions meet the stability range, scheduling is allowed to be triggered again; if the situation remains volatile, the current strategy is maintained. Through this rhythmic scheduling mechanism, the system achieves a dual balance between memory pressure relief and numerical stability.

[0035] Dynamic cooldown period length: , :step Cooling period length (unit: number of steps). : Shortest cooling length The amplification factor of fluctuations to cooling. : Fluctuation measurement window size :step The observed peak memory, :step Observed training loss.

[0036] Criteria for allowing switching: Smooth updates: , :step Allow scheduling switchover? : The step number in which the last handover occurred : Candidate optimal triplet. : Maximum change limit for a single event (to avoid jumps). :step Smoothed triplet, Smoothing coefficient (smaller when the fluctuation is greater). , : Upper limit and base value of smoothing coefficient Fluctuation penalty gain :step The comprehensive volatility index (which can be composed of the weighted standard deviation of memory / loss / throughput).

[0037] The batch cumulative scheduling module maintains close communication with the memory monitoring and budgeting module during system operation. When the memory monitoring and budgeting module issues a memory pressure signal, the batch cumulative scheduling module immediately initiates the three-domain coupled scheduling logic, automatically reducing the micro-batch size and increasing the cumulative steps. Simultaneously, it synchronizes the current batch adjustment strategy to the gradient adaptive pruning module, ensuring the pruning threshold matches the new batch configuration. Once the memory state returns to a safe zone, the module gradually relaxes the micro-batch restrictions and extends the sequence length, achieving a dynamic balance between memory utilization and training efficiency. The batch cumulative scheduling module interacts with the training convergence monitoring module, which provides feedback on the batch adjustment effect based on the convergence score. If the convergence score decreases or fluctuates beyond a threshold, the system rolls back to the previous stable batch configuration. Through closed-loop control among multiple modules, the dynamic adjustment of the batch cumulative scheduling module responds to memory pressure while maintaining training stability. The batch cumulative scheduling module achieves an adaptive balance between memory load and computational efficiency through three-domain coupled scheduling technology, and through self-game exploration technology, it allows the batch parameter combinations to continuously evolve during training, possessing self-optimization capabilities without manual intervention. Cross-node heterogeneous shaping technology enables the system to run stably in heterogeneous hardware environments, preventing memory bottleneck nodes from becoming training bottlenecks. Window order preservation and cooling mechanisms ensure the smoothness of the scheduling strategy adjustment process, preventing frequent parameter switching from interfering with the convergence process.

[0038] The recomputation and sharding optimization module achieves memory compression and resource optimization in both time and space dimensions through activation recomputation, parameter sharding, and pipelined execution mechanisms. It receives memory budget signals from the memory monitoring and budgeting module and dynamically determines the recomputation ratio, sharding granularity, and pipelined scheduling method based on the memory risk level, thereby maintaining the complete training process of the model under memory constraints. The recomputation and sharding optimization module, along with the gradient adaptive pruning module and the batch cumulative scheduling module, forms a memory control link, and together with the loss scaling protection module and the training convergence monitoring module, forms a collaborative closed loop of numerical and convergence constraints. Fine-grained critical path recomputation technology is used to selectively enable recomputation in the computation graph to reduce memory dependence on the activation cache. During the training initialization phase, the system performs topological analysis on the model computation graph to determine the memory consumption of each forward operator and its sensitivity to the final gradient. Through this analysis, the system generates a critical path graph, marking subgraph nodes with high memory consumption but low gradient impact. During actual training, recomputation is only enabled for these critical path nodes, meaning the forward activation values ​​are recalculated during backpropagation, without retaining the entire activation cache in memory. The system assigns an independent recomputation flag and caching strategy to each recomputation node, ensuring that the recomputation process is executed separately from normal forward computation to avoid interfering with the main computation flow. The recomputation ratio is dynamically adjusted based on the memory budget signal. When memory is in a safe zone, the recomputation ratio is maintained at a low level to ensure throughput efficiency; when memory enters a stress zone, the recomputation range is automatically expanded to compress activation storage peaks. Fine-grained critical path recomputation also features a "layer-by-layer update" characteristic, re-evaluating the set of critical path nodes based on the new memory curve at each training stage, thereby maintaining the timeliness and adaptability of the recomputation strategy.

[0039] Node recalculation priority score: , In training steps For computation graph nodes The recalculation priority score (the higher the score, the higher the priority for recalculation). :node Video memory (bytes) required for forward activation cache. The maximum value of activated memory for all nodes in this model (normalized scale). :node Gradient sensitivity metric (can be fitted using a local second-order approximation or loss change ratio; a larger value indicates a greater impact on the gradient) : In all nodes The maximum value (normalized scale). :node Time (in seconds) required for forward recalculation. : In all nodes The maximum value (normalized scale). The weighting coefficients of the three factors (nodes favored for "high memory usage, low sensitivity, and low latency" are recalculated). Recalculation ratio: Budget linkage: , :step The percentage of nodes that will be recalculated (selected from highest to lowest score). Baseline recalculation ratio (default coverage of the safe zone). The amplification factor from pressure to proportion. Standardized pressure intensity (relative magnitude after exceeding the safety limit). The memory stress index in the next budget signal (the normalized strength of the next peak relative to the upper limit). :step Risk label : The safety threshold corresponding to the risk zone (e.g., 0.8 for the safe zone, 0.7 for the tense zone, and 0.6 for the over-limit zone, which can be set by engineering). : Upper limit of recalculation ratio to prevent excessive recalculation, selection set: in Step-by-step score highest Each node forms a recalculation set. Activate video memory: The effect of recalculating the overhead: , :step The approximation of the video memory (bytes) corresponding to the activated cache under the recalculation strategy. : Minimum retention factor for recalculated nodes (only indexes / boundaries are retained, significantly less than 1). :step Additional time (seconds) introduced by recalculation. The recalculation multiple during the reverse propagation phase (for some operators, 1-2 forward propagations are required to restore activation). In this way, the system achieves continuous optimization of GPU memory usage within a controllable computational overhead. The three-layer fragmentation peak-shifting technique distributes peak GPU memory usage across the time dimension through a layered loading and unloading strategy. Model parameters and optimizer states are divided into three fragmentation levels: layer group fragmentation, tensor block fragmentation, and time slice fragmentation. Layer group fragmentation is the top-level segmentation method, dividing the model into multiple layer groups according to the hierarchical relationship of the network structure, with each layer group corresponding to a set of consecutive network layers. During forward and backward propagation, the system only loads the parameters of the current layer group into GPU memory, and immediately unloads them to main memory or the communication cache after the layer group's computation is complete, thus avoiding the simultaneous resident of all model parameters. Tensor block fragmentation divides the high-dimensional parameter matrix within a layer group, splitting large tensors into several smaller blocks. A GPU memory mapping table is established for each tensor block to ensure that only the necessary block region is loaded each time to support the current operator's computation. This significantly reduces the amount of GPU memory required for a single computation. Time-slice partitioning addresses the training time dimension by asynchronously loading and writing back optimizer states (such as momentum or second-order cumulative terms) in time slices. The system uses scheduling control to load the optimizer states of the next layer in parallel during the backpropagation of the previous layer, achieving time overlap. The coordinated operation of the three-layer partitioning, with staggered peak times, ensures that activation peaks and parameter peaks are separated by time windows, thus smoothing the memory curve. The scheduler monitors the loading and unloading sequence of each layer. When the memory budget signal indicates a risk of exceeding limits, the scheduler shortens the residence time of each layer and elevates the partition level, thereby completing the memory compression operation within a millisecond-level response time.

[0040] The segment dwell ratio is similar to the "peak value after peak shifting": , :step The total peak video memory is approximately (in bytes). :step Activate the relevant video memory. :step Average co-occurrence ratio of layer fragments (the percentage of time each layer spends simultaneously within each cycle). :step The parallel residency ratio of tensor block fragments (the proportion that still needs to reside in parallel after block-level partitioning). :step The co-occurrence ratio of time slices (the average residence rate of optimizer states over time). Total memory usage for all model parameters (total bytes without fragmentation). : The average number of bytes of block memory required per cycle after block-level partitioning. : Total memory for optimizer state (e.g., momentum / second-order terms), three The value is determined by the scheduler's "load-compute-unload" sequence and is actively reduced as the risk level increases to achieve peak shifting. The scheduling sequence definition for the residency ratio is as follows: , , Duration of a single data segmentation scheduling cycle (in seconds). Number of layers. :step Layers Duration of stay (in seconds) within a cycle. Number of parameter blocks :step piece Duration of stay : Number of state time slices :step Time slice Duration of stay. Risk-controlling law for the stay ratio: , , , Baseline co-occupancy ratio at each level (occupancy level when throughput is prioritized). Minimum concurrent ratio for each level (to prevent excessive splitting from causing jitter). The slope of the suppression from pressure to residence ratio. Normalized stress intensity. This mechanism ensures that the model can maintain the continuity of the training process even under dynamic memory pressure.

[0041] The compression-communication-computation triple pipeline technique maximizes hardware utilization while optimizing GPU memory. It constructs an asynchronous pipeline for the three types of operations involved in training: parameter compression, cross-GPU communication, and backpropagation, achieving parallelization through an event-driven mechanism. During the backpropagation computation phase, the GPU core performs gradient calculations, while CPU threads or independent DMA channels simultaneously initiate parameter compression operations, compressing and encoding the calculated parameter blocks and temporarily storing them in main memory or a communication buffer. The compression process is completely parallel to the backpropagation, thus hiding the compression overhead. Once compression is complete, the communication thread immediately initiates cross-GPU transmission, sending parameter or gradient data to other devices for synchronous updates. Simultaneously, the GPU continues executing the next batch of backpropagation, achieving time overlap between communication and computation. An event synchronization mechanism ensures data consistency; when compression and communication operations are complete, the system triggers a synchronization signal, allowing the computation flow to proceed to the next stage. The advantage of this triple pipeline structure is that it decomposes the originally serially executed memory compression, communication, and computation processes into independent pipeline stages, ensuring that memory release, data exchange, and computation do not block each other. The system automatically updates the buffer state at the end of each pipeline cycle to ensure that the video memory is always in an optimal allocation state, thereby significantly reducing peak usage and improving throughput efficiency.

[0042] Phased delay model for segmented pipelines: , : Sample / Token size of a single pipeline block : Block data size (bytes) For fixed metadata, The byte coefficient per unit size Compression throughput (bytes / second) Compression ratio (bytes after compression / bytes before compression) Link bandwidth (bytes / second) Network end-to-end fixed latency (seconds). Forward time (in seconds) for this block. : The time ratio coefficient between the backward and forward directions. : Compress / communication and reverse overlap efficiency (1 indicates complete hiding). Effective stage bottleneck latency (steady-state cycle time of each block) under multi-stage parallelism. Pipeline throughput: Filling amortization: , Number of blocks in one reverse cycle Total time of the cycle (seconds). Pipeline filling / draining overhead (seconds), typically varies. Increases while relatively decreasing The number of valid samples / Tokens processed in this cycle (based on the block size) and (Determined by the number of blocks) Steady-state latency from 3.1. Instantaneous effect of pipeline on memory release (compression → unloading): , :step The video memory release rate per unit time (bytes / second) is approximated by the rate at which the compressed video memory is moved out. Release efficiency coefficient (considering pipeline scheduling and DMA contention). Compression ratio , : Block size in bytes / stage latency : Observation window length (seconds), used to approximate peak reduction within this window. :step Peak memory usage is approximate. Sample-level recalculation and lightweight caching techniques are used to locally optimize the uneven memory distribution within micro-batches. Due to significant differences in input sequence length, branch complexity, and activation paths among different samples, some samples may experience excessive memory usage during forward propagation. By monitoring the memory usage of each sample in real time, high-occupancy samples exceeding the average threshold are identified, and a local recalculation strategy is applied to these samples individually. In local recalculation, the system only retains the input tensor and intermediate index information of these samples, without caching complete activation values, and recalculates the forward propagation results during backpropagation. For samples with normal memory usage, the system uses a lightweight cache structure for storage. The lightweight cache area adopts a circular design, storing only the activation data from the two most recent training steps, and automatically overwriting old data with each update to prevent cache accumulation and memory waste. During execution, the sample-level recalculation ratio is dynamically adjusted based on the memory budget signal. When memory is in a safe zone, recalculation is only enabled for extreme samples; when memory enters a tense zone, the local recalculation range is expanded, and the lightweight cache capacity is reduced to further compress peak memory usage. The introduction of sample-level recalculation enables the system to achieve fine-grained control of memory allocation while maintaining training accuracy.

[0043] "Double Sample" Judgment: Recalculation probability: , :step sample Forward activation memory requirements (bytes). , :step The moving mean and standard deviation of the sample activated memory. Numerical stability constant :step sample Standardized occupancy (Z-Score). ReLU main part, Re-sample threshold (exceeding the threshold is considered high occupancy). :step sample The probability of triggering recalculation. : Probability slope (the higher the Z-score, the more likely it is to be recalculated). The overall sample recalculation share is linked to the budget: , :step The proportion of samples that can be recalculated in a microbatch Baseline sample recalculation ratio The amplification factor from pressure to sample proportion. Standardized pressure intensity : Upper limit of sample recalculation ratio. For a micro-batch, according to Choose from largest to smallest, no more than A portion of the samples enables local recalculation. Lightweight cache capacity: , budget linkage: , : step The number of steps retained by the circular lightweight cache (0 - 2 steps, typically 2 steps, automatically reduced to 1 or 0 under high pressure), : The maximum number of steps that can be retained, : The reduction coefficient from pressure to cache steps, : step The cache occupancy ratio coefficient of normal samples (the conversion ratio of lightweight cache to full activation), : The minimum retention coefficient of recalculated samples (only retain input and index), : step The set of samples judged as heavy samples, : step Approximation of effective activation video memory at the sample level.

[0044] The recalculation sharding optimization module maintains real-time linkage with the video memory monitoring budget module. The video memory monitoring budget module transmits the video memory usage status through the budget signal. When it detects that the video memory occupancy enters the critical area, the recalculation sharding optimization module will automatically adjust the recalculation ratio and sharding level to gradually reduce the video memory peak. When the video memory returns to the safe range, gradually resume the normal calculation mode and reduce the recalculation frequency to improve the throughput efficiency. The recalculation sharding optimization module runs in coordination with the batch accumulation scheduling module. When the batch accumulation scheduling module reduces the micro-batch due to video memory pressure, the recalculation sharding optimization module synchronously reduces the sharding level and recalculation ratio, making the video memory release and batch adjustment form a complementary balance. The communication between the recalculation sharding optimization module and the training convergence monitoring module ensures that the video memory optimization process does not affect the convergence performance. When the convergence score drops or the value is unstable, it will automatically roll back to the previous stable sharding configuration to prevent the video memory optimization from introducing training oscillations. Through the above multi-layer coordination mechanism, the recalculation sharding optimization module achieves the balance between video memory peak shaving and calculation continuity, enabling the system to have the ability to adaptively adjust under different video memory constraint conditions. The recalculation sharding optimization module directly determines the change trend of the video memory peak in the entire system. Through the fine-grained critical path recalculation technology, it reduces the video memory occupancy while retaining the core activation information; through the three-layer sharding peak staggering technology, it staggers the parameter and activation peaks in the time dimension to achieve resource smoothing; through the compression - communication - calculation triple pipelining technology, it realizes the parallelism of communication, compression and calculation at the performance level; through the sample-level recalculation and lightweight cache technology, it realizes the refinement and personalization of video memory optimization at the micro-batch level.

[0045] The loss scaling protection module serves as the numerical safety layer of the entire system, ensuring the stability of numerical computation during mixed-precision training and preventing training interruptions due to gradient overflow, aberration amplification, or the appearance of non-numerical values ​​(NaN). It dynamically adjusts the loss scaling coefficient by continuously monitoring the changing trends of the loss value and gradient distribution during training, forming a closed loop with the memory monitoring budget module and the batch accumulation scheduling module. In the overall system flow, the loss scaling protection module intervenes before and after each backpropagation, ensuring that the numerical accuracy under mixed precision is not compromised and that the training process does not experience anomalies due to extreme gradient fluctuations through mechanisms such as early warning, rollback, recovery, and policy linkage. Multi-granularity early warning mapping technology is a fundamental component of the loss scaling protection module, aiming to achieve early identification of numerical anomalies through layered detection. The system deploys monitoring probes at three levels: operator-level, layer-level, and global-level. Operator-level probes record the gradient distribution characteristics of each operator during backpropagation, including the proportion of extreme values, the rate of change of the mean, and variance fluctuations. Layer-level probes monitor the overall gradient distribution of each layer in the network to identify phenomena such as concentrated bursts or vanishing local gradients. The global probe aggregates monitoring results from all layers to generate a training stability heatmap. The heatmap uses time step as the horizontal axis and gradient statistics for each layer as the vertical axis, with color intensity reflecting the degree of anomaly. Based on the dynamic trends of the heatmap, the system calculates the risk level of the current training state. When the heatmap shows an abnormal concentration of gradients or a continuous increase in gradient variance in a certain layer, that layer is immediately marked as a high-risk area, and an early warning mechanism is activated. During the early warning phase, the loss scaling parameters are not immediately adjusted; instead, an observation period is initiated to confirm whether the anomaly persists. Only if the anomaly persists for multiple training steps does the system trigger subsequent rollback operations. The core of this technology lies in the early detection of numerical anomalies, allowing loss scaling adjustments to intervene before risk accumulation, rather than relying on the final overflow signal, thus significantly reducing the probability of training interruption.

[0046] Operator-level anomaly strength and risk score: , : at time step Operator Risk score, The weighted coefficients of the three terms: extreme values, mean drift, and sudden increase in variance. Operator Extreme value ratio (the proportion of gradient samples that go out of bounds). Operator Relative mean shift Operator The variance increased relatively suddenly. :step Operator No. Gradient sample values, :step Operator The number of samples, Extreme value threshold coefficient (unit: "multiple of standard deviation") , :step Operator The gradient mean / standard deviation, Numerical stability constant Indicator function (1 if true, 0 otherwise). Hierarchical aggregation: Updated global heatmap: , :step layer The average risk level of the hierarchy. :layer The set of operators contained therein :step Heat map in layers The value (smoothed hierarchical risk). Time smoothing coefficient. Risk level determination: , , :step Global risk level (green / yellow / red). :step Global layer average heat value, Number of floors The tiered threshold (the boundary between the observation period and the warning) tiered rollback – progressive recovery technology is the core means for the module to deal with numerical anomalies, and is used to safely and smoothly restore a stable training state after an anomaly is detected.

[0047] When an overflow or abnormal amplification occurs in a certain layer, a local rollback operation is first performed, reducing the loss scaling factor only for the layer or operator where the anomaly occurred, so that the gradient calculation of that layer immediately returns to a safe range. If the abnormal layer still has not recovered after several consecutive training steps, a global rollback operation will be triggered, that is, the loss scaling factor of the entire model will be reduced simultaneously. Both local and global rollbacks have dynamic cooling times during execution to prevent frequent switching of scaling states in a short period of time. The number of stable steps after each rollback is recorded. When the number of consecutive stable steps reaches a preset threshold, the system enters the incremental recovery phase. Incremental recovery adopts a small-step incremental strategy, increasing the scaling factor by only a very small amount each time, and observing whether the training stability is maintained. When the system detects that the gradient variance and the rate of change of loss are both within the normal range, the recovery phase ends, and the loss scaling factor returns to the initial target value. Through this strategy of combining layering and incremental approaches, the system can flexibly handle local anomalies without affecting the global training progress, achieving a smooth transition from anomaly detection to stable recovery, and avoiding repeated training interruptions caused by numerical oscillations.

[0048] Partial rollback trigger: Zoom update: , :step layer The local loss scaling factor, :step layer The current scaling factor, Local rollback rate :layer The overthreshold amplitude (normalized intensity exceeding the threshold). Hierarchical rollback threshold : Continuous abnormality indication (1 if there has been a continuous high risk in recent steps). :step layer Heat value. Global rollback triggered: Zoom update: , :step The global loss scaling factor, :step The current global scaling factor, Global rollback rate Global over-threshold amplitude :step Global layer average heat value, Global rollback threshold. Gradual recovery: , , :step The scaling factor after restoration :step The current scaling factor (local or global, depending on the operating mode). : Target upper limit scaling (target value during initialization). :step Restore step size, : Restore the step size base value, : Coefficient of variation suppresses gain, :step coefficient of variation of gradient ( and (Global gradient standard deviation / mean) :step Abnormal intensity index (optional) (or its monotonic function). : Entering the safe threshold for recovery, :step Continuous stable steps (count of the most recent consecutive steps without anomalies). The minimum number of stable steps required to enter recovery. : Numerical stability constant. The policy-linked guardrail technology coordinates the dynamic interaction between the loss scaling protection module and other key modules, ensuring that adjustments by multiple modules do not conflict within a numerically safe range. Since changes in loss scaling directly affect gradient magnitude, and the gradient is controlled by the adaptive gradient pruning module, while the memory state is associated with the batch accumulation scheduling module, policy resonance may occur if multiple modules adjust simultaneously. The system incorporates bidirectional guardrail logic in the loss scaling protection module. When the loss scaling factor is in a contracted state, a synchronization signal is sent to the adaptive gradient pruning module, instructing it to temporarily increase the pruning intensity to prevent gradient oscillations caused by reduced scaling; simultaneously, a freeze command is issued to the batch accumulation scheduling module, temporarily locking the micro-batch size and accumulation steps to prevent amplified memory fluctuations. When the loss scaling factor recovers, the restrictions are lifted in the order of first loosening the pruning and then widening the batch size, ensuring that the entire system gradually returns to its original equilibrium state during the recovery phase. The policy-linked guardrail mechanism guarantees the consistency and coordination between loss scaling adjustment, memory optimization, and gradient control, preventing parameter coupling misalignment or numerical oscillations during dynamic adjustments, thus maintaining the continuity and stability of overall training.

[0049] Resonance Risk Measurement: Freeze gate: , :step Strategy resonance risk measurement :step Scaling changes :step Micro-batch changes, :step Average cropping threshold variation ( (Weighted average of segment thresholds) Three weights Resonance trigger threshold Freeze gating (1 indicates that the batch / cumulative changes are reset to zero for one cooldown period). Double-sided guardrails: , : Safe upper and lower bounds for the effective gradient scale :step Loss scaling factor :step The clipped gradient vector : The empirical mean for a short window : Trimming reference scale (such as the average threshold at initialization). Numerical stability constant. Drift adaptive memory technology is used to improve the system's response efficiency to repetitive numerical anomalies.

[0050] Throughout the training process, the system continuously records the numerical profile of each anomaly, including the anomaly layer number, gradient statistical features, memory usage status, and loss scaling trajectory. This information forms a historical anomaly sample library for subsequent judgment. When the system detects a new numerical anomaly, it compares the current anomaly profile with the records in the historical sample library. If the similarity exceeds a set threshold, it indicates that the anomaly type is consistent with historical cases. The system does not need to re-enter a lengthy warning and recovery process but directly loads the corresponding conservative scaling template. The conservative template contains validated stable scaling coefficients and corresponding cooldown times, and can take effect immediately, thus quickly restoring the training to a stable state. The sample library is continuously updated during long-term training, and the recovery process for each anomaly is evaluated. If a certain anomaly type is quickly corrected after multiple occurrences and does not recur, the priority of that template is automatically reduced to minimize unnecessary conservative intervention.

[0051] Abnormal profile: Template similarity: , :step Anomaly profile vector (including thermal intensity, gradient statistics, scaling changes, and memory pressure). :step Global layer average heat value, , :step Global gradient standard deviation / mean :step Scaling change :step Memory stress index (derived from memory budget signal; the higher the value, the higher the stress). : Image vector of historical anomaly template :step With template cosine similarity, Inner product and L2 norm.

[0052] The drift-adaptive memory technology enables the system to have self-learning capabilities, continuously improving protection response efficiency through historical experience and achieving the evolution of numerical safety strategies. The operation logic of the loss scaling protection module can be divided into four stages: monitoring, early warning, rollback, and recovery. The monitoring stage continuously constructs and updates multi-granularity early warning maps; the early warning stage enters an observation period when an upward trend in risk is detected; the rollback stage performs local or global scaling adjustments based on the severity of the anomaly; and the recovery stage uses gradual improvement and guardrail coordination to achieve smooth regression. Throughout the operation, the loss scaling protection module maintains high-frequency interaction with the gradient adaptive pruning module, keeps parameters synchronized with the batch accumulation scheduling module, and feeds back the numerical safety status to the training convergence monitoring module for evaluating changes in the convergence score. When the memory monitoring budget module issues a memory shortage signal, the loss scaling protection module appropriately reduces the scaling factor to reduce memory load fluctuations caused by instantaneous gradient amplification. If the gradient adaptive pruning module detects gradient anomalies during this period, the system will prioritize scaling adjustments by the loss scaling protection module rather than further increasing the pruning force to prevent effective gradient loss. Through this collaborative relationship, the system achieves dual stability control of memory and numerical values. The loss scaling protection module achieves end-to-end control over numerical anomalies through technologies such as multi-granularity early warning, hierarchical rollback, policy guardrails, and drift memory. Its performance in mixed-precision training demonstrates that, when running in conjunction with other modules, the loss scaling protection module further enhances the overall robustness of the system. Gradient pruning and loss scaling complement each other, ensuring consistency between gradient magnitude control and numerical scaling; batch scheduling and scaling adjustment work together to achieve unified optimization of memory usage and numerical stability.

[0053] The training convergence monitoring module serves as the intelligent decision-making and coordination center of the entire system. It provides unified monitoring, analysis, and policy coordination for memory status, gradient dynamics, batch configuration, recomputation behavior, and numerical stability on a global scale. It records the operational status and interactions of each module, performs causal analysis and priority self-evolution of system policies, thereby forming a self-learning and self-adjusting global control mechanism. At different stages of system operation, the training convergence monitoring module plays two roles: in the real-time training phase, it acts as a decision controller, responsible for integrating memory budget signals, gradient features, and convergence metrics, and dynamically allocating policy weights; in the post-training evaluation phase, it acts as a data analyzer, responsible for generating convergence scores and interpreting results, and using these to guide subsequent policy optimization. Causal trajectory modeling technology is used to record and analyze the causal relationships between policy actions and results during training, and is the core foundation of the training convergence monitoring module. For each policy adjustment, a four-tuple of "action-context-result-control" is established. "Action" refers to the specific operation taken by the system at a given moment (e.g., adjusting pruning intensity, changing batch size, or switching recalculation mode); "context" includes memory state, gradient stability, and the convergence rate at the current stage; "result" refers to the change in convergence speed or numerical stability after the action; and "control" is the historical baseline performance without adjustment. By continuously accumulating this causal event data, a complete causal trajectory sequence is formed. Each trajectory includes information such as timestamp, acting module, policy magnitude, convergence response, and memory feedback. The system uses this trajectory record to analyze the effects of different policy combinations under different memory and gradient conditions, thereby identifying the causal pattern of "which policy is optimal under what conditions." When the system enters a new training phase or the data distribution changes, it can quickly match the most similar scenario in the historical trajectory and prioritize the use of previously validated policy combinations.

[0054] , :step Select Action Contrast action Causal gain estimation, :step The actual action label performed : Indicator function, executed Select 1, otherwise 0. :step The context feature vector (including memory pressure, gradient stability, batch / sequence configuration, scaling state, recent convergence speed, etc.). :step The observation results (such as the decrease in loss per unit time or the incremental score in stages). :step In context Select action Probability estimation of the tendency to dispose of, :step In context Adopt action Expected results estimation :step In context Adopt the comparison action The expected result is estimated.

[0055] The continuous operation of causal trajectory modeling technology enables the system to learn from its own behavioral outcomes, allowing it to infer future decision-making directions from historical behavior, thus achieving self-learning and self-evolution of policy decisions. The policy priority self-evolution technology, built upon the results of causal trajectory modeling, is used to dynamically adjust the policy execution order and adjustment magnitude of the six modules within the system. The training convergence monitoring module performs statistical analysis on the causal trajectories, calculating the contribution of each module's actions to past training gains. Gain is defined as the combined value of the convergence score improvement and memory stability improvement caused by the action. Dynamic priorities are assigned to each module based on the gain contribution. When a module's policy exhibits high gains without negative impacts in multiple training iterations, its priority is automatically increased, granting it greater policy weight and adjustment frequency in the next training round; conversely, if multiple policy adjustments cause numerical oscillations or convergence delays, its priority is reduced, or even temporarily frozen. Priority adjustment follows the principle of "rapid local response, smooth global transition," meaning only minor changes are allowed in a single training step, accumulating over multiple training phases to form a significant self-evolutionary effect. Through this mechanism, the system gradually forms an optimal strategy cooperation pattern, so that the effects of memory control, gradient stabilization and numerical protection can reach a dynamic balance.

[0056] , Next step module The execution priority weight (the larger the weight, the earlier it will be scheduled). Current step module Execution priority weight, Time smoothing coefficient (limits single-step transitions) :step Module The overall benefit score (a scalar weighted by convergence improvement, numerical stability improvement and memory impact). Temperature parameter (the smaller the value, the more "greedily" it tends to favor higher returns). Normalize and sum all candidate modules.

[0057] The policy priority self-evolution technique ensures that the system can continuously self-optimize during long-term operation without human intervention. The interpretable convergence scoring technique is used to quantitatively evaluate the overall convergence quality of the system during training and provides traceable scoring components and explanations. After each training cycle, a unified convergence score is generated by comprehensively calculating convergence speed, stability, effective gradient ratio, and generalization surrogate metric. Convergence speed represents the rate of decrease of the loss function per unit time; stability represents the magnitude of gradient fluctuations and the frequency of numerical safety events; the effective gradient ratio measures the proportion of gradients that still participate in parameter updates after pruning and scaling; and the generalization surrogate metric reflects the performance trend of the model on the validation set. These metrics are weighted and summarized to generate a stage-specific convergence score, which is compared with historical scores to determine the merits of the current policy combination. A score explanation report is output, explaining the reasons for score changes. For example, when the system detects that "the pruning intensity of the adaptive gradient pruning module is too high, leading to a decrease in the effective gradient ratio" or "the batch cumulative scheduling strategy is too conservative, resulting in a slowdown in the convergence rate," the report will clearly indicate the source of the problem and directions for improvement.

[0058] , :step The convergence score, :step and Training loss, : Length of the scoring window (seconds or steps) :step Frequency of numerical anomalies (e.g., overflow / NaN count per unit time). :step Global gradient standard deviation / mean :step The effective gradients (scaled and clipped) that participate in the update. :step Gradient before scaling / clipping :step and Validation metrics (such as accuracy or its monotonic transformation). Four weighting coefficients Numerical stability constant.

[0059] This interpretable mechanism enables the system to autonomously identify unreasonable policy parameters and automatically correct them during subsequent training. Through interpretable convergence scoring technology, the system's training state is no longer a black box, but a quantifiable, diagnosable, and optimizable process. This mechanism provides a scientific basis for decision-making regarding the long-term stable operation of the system. Gray-scale release and safe rollback technologies are key mechanisms for policy verification and risk control in the training convergence monitoring module. When the system generates a new policy combination (e.g., adjusting the parameter linkage logic of multiple modules), the training convergence monitoring module does not immediately apply it globally, but instead uses a gray-scale release approach, enabling it locally on some nodes or small batches of tasks. During the gray-scale phase, the system continuously monitors the convergence score, memory usage, and numerical fluctuations, comparing them with the performance of the original stable policy. If the new policy shows an improvement in convergence score without causing memory fluctuations during the gray-scale phase, the training convergence monitoring module promotes it to global execution; if a score decrease or fluctuation exceeds a preset threshold, the system immediately triggers a safe rollback mechanism, restoring the previous stable policy configuration. The rollback process is completed in milliseconds without interrupting the training process, and at the same time the triggering reason, affected modules and score difference are recorded so as to avoid the recurrence of similar combinations in subsequent policy evolution.

[0060] , :step Determining whether to expand the gray-scale promotion (1 = promotion, 0 = no promotion). :step The Z-statistic of the new strategy relative to the baseline (the ratio of the variance-normalized difference between the two sets of scores). Significance level The normal quantile threshold, :step Risk indicator (a scalar measure that combines memory fluctuations and the frequency of numerical events). Risk ceiling threshold (if exceeded, it will not be promoted). :step The average convergence score of the new / baseline strategy within the grayscale range. Minimum acceptable improvement : Indicator function (1 if the condition is met, 0 otherwise).

[0061] The canary release and safety rollback mechanism ensures the safety and stability of the system when exploring new strategies, allowing strategy optimization to be conducted in a trial-and-error manner without posing a risk to the overall training process. This mechanism endows the system with the ability to "train, validate, and correct simultaneously," achieving compatibility between stability and innovation. The training convergence monitoring module occupies the highest-level control position in the entire system, running throughout the entire training process. After each training step, real-time runtime data is collected from the memory monitoring budget module, gradient adaptive pruning module, batch cumulative scheduling module, recalculation sharding optimization module, and loss scaling protection module. This data includes memory budget values, gradient distribution status, batch configuration parameters, sharding loading time, and numerical anomaly records. This data is integrated into a unified training state vector, input into the causal trajectory modeling unit, and the causal relationship table is updated in real time. When a decrease in convergence score or abnormal memory fluctuation is detected, the strategy priority self-evolution process is immediately executed, making local adjustments to the weight allocation of each module. If the abnormal trend persists, the canary release mechanism is triggered to verify the effect of the new strategy combination in a small number of nodes. Once the new strategy is validated, it is then globally promoted, achieving incremental system optimization. The operational logic forms a complete loop: data acquisition, causal modeling, priority adjustment, policy verification, and scoring feedback, constituting a self-learning and self-evolutionary closed loop for the system. The training convergence monitoring module achieves system-level intelligent policy learning through causal trajectory modeling, enabling modules to no longer operate independently but to collaboratively optimize under a unified decision-making logic. Policy priority self-evolution technology allows the system to gradually form a stable and efficient policy distribution during long-term training, significantly reducing reliance on manual parameter tuning. Explainable convergence scoring technology gives the system self-diagnostic capabilities, enabling it to identify the causes of training performance degradation in real time and propose adjustment suggestions. The canary release and safe rollback mechanisms ensure the system's security and robustness during policy updates, ensuring that every adjustment is within a controllable range. Through the overall coordination of the training convergence monitoring module, the system as a whole achieves a shift from passive control to active evolution. In long-term operation, it can continuously improve the system's stability, convergence speed, and numerical robustness, enabling large models to maintain efficient convergence even in memory-constrained hardware environments.

[0062] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A large model gradient pruning training optimization system that reduces GPU memory usage, characterized in that: It includes the following six functional modules: The memory monitoring and budget module is used to collect and statistically analyze the memory usage during the training process in real time, predict the memory peak value in the next training step, and generate a memory budget signal. The gradient adaptive pruning module is used to perform multi-segment adaptive pruning of the model parameter gradients based on the memory budget signal, so as to control the gradient magnitude and maintain the stability of the gradient distribution. The batch accumulation scheduling module is used to dynamically adjust the micro-batch size, gradient accumulation steps, and sequence length limit under the constraints of the video memory budget, so as to achieve a balance between video memory and computing throughput. The recalculation and sharding optimization module is used to perform critical path recalculation based on the computation graph structure, and to shard and stagger the loading of parameters and optimizer states to reduce peak memory usage. The loss scaling protection module is used to dynamically adjust the loss scaling factor during mixed-precision training to prevent gradient overflow and maintain numerical stability. The training convergence monitoring module is used to perform causal analysis and priority adjustment on the running status of each module, calculate the convergence score and form policy feedback, and establish a system self-learning closed loop. The six modules communicate with each other through memory budget signals, training feedback signals, and policy instructions, thereby achieving a collaborative closed loop of memory usage control, gradient stabilization, and training efficiency optimization.

2. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The memory monitoring and budgeting module includes: memory sampling and hierarchical statistical technology for collecting memory usage, remaining capacity, and operator-level percentage of each computing device through the memory interface, and dividing the memory information into model parameter memory layer, optimizer state layer, activation cache layer, and communication buffer layer; memory trend prediction technology for predicting the next memory peak based on historical memory curves and batch configuration; risk band constraint technology for dividing the memory state into safe zone, tense zone, and over-limit zone based on the prediction results, and sending adjustment signals to subsequent modules when entering the tense zone; and memory feedback and correction technology for calculating prediction errors and adjusting prediction parameters to achieve self-learning and correction of the memory budget.

3. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The gradient adaptive pruning module includes: a semantic segment pruning technique for dividing parameters into different semantic segments based on the model structure and setting independent pruning rules for each segment; a step stabilization window technique for calculating the gradient mean, variance, and peak frequency within a fixed-length window, generating a steady-state score, and adjusting the pruning intensity; a noise consensus gating technique for synchronously judging abnormal gradients across multiple computing nodes, triggering global suppression only when a majority of nodes consistently detect an anomaly; and a topological affinity suppression technique for identifying strongly correlated parameter groups in the computation graph and maintaining their gradient ratios to prevent local oscillations.

4. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The batch cumulative scheduling module includes: a three-domain coupled scheduling technique that uses the micro-batch size, gradient accumulation steps, and sequence length as joint scheduling parameters to perform synchronous adjustment based on the memory status; a self-game exploration technique that constructs a pool of candidate parameter combinations, rotates experiments in the short term, and retains the optimal combination based on the payoff and risk scores; a cross-node heterogeneous shaping technique that allocates the load shape based on the memory capacity and bandwidth capabilities of each device in a multi-computing environment to achieve heterogeneous synchronization; and a window order preservation and cooling-off technique that sets a policy cooling-off period and a freeze period to limit frequent switching and ensure the stability of the training process.

5. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The recomputation and fragmentation optimization module includes: fine-grained critical path recomputation technology to identify subgraphs with high memory usage and low gradient sensitivity, and perform activation recomputation only on these subgraphs to reduce memory usage; three-layer fragmentation peak-shifting technology to divide parameters and optimizer states into layer fragmentation, tensor block fragmentation, and time slice fragmentation, and load and unload them in different time windows; compression, communication, and computation pipelining technology to perform parameter compression, communication transmission, and reverse computation in parallel during the computation process to achieve resource peak-shifting; and sample-level recomputation and lightweight caching technology to perform local recomputation on samples with high memory usage, while other samples retain lightweight caching to reduce peak memory usage.

6. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The loss scaling protection module includes: multi-granularity early warning mapping technology for deploying probes at the operator, hierarchical, and global levels to generate stability heatmaps and detect numerical anomalies; hierarchical rollback and progressive recovery technology for performing local scaling rollback when anomalies occur in a local layer, performing global rollback when necessary, and restoring the scaling coefficient in small steps during the recovery phase; strategy linkage guardrail technology for establishing linkage logic with the gradient adaptive pruning module and the batch accumulation scheduling module to control the parameter adjustment order during scaling contraction and recovery; and drift adaptive memory technology for recording abnormal numerical characteristics and loading a conservative scaling template when a similar distribution is detected.

7. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The training convergence monitoring module includes: causal trajectory modeling technology for recording the correlation events between memory changes, policy actions, and training results; policy priority self-evolution technology for dynamically adjusting the policy priority and amplitude range of each module based on causal benefit statistics; interpretable convergence scoring technology for calculating the convergence score based on convergence speed, stability, effective gradient ratio, and generalization index, and outputting the reasons for score changes; and canary release and safe rollback technology for partially enabling and monitoring the convergence score when a new policy is introduced, and rolling back to the previous stable configuration if the score drops below a preset threshold.

8. The large model gradient pruning training optimization system for reducing GPU memory usage according to claim 1, characterized in that: The memory budget signal output by the memory monitoring budget module is used by the gradient adaptive pruning module, the batch cumulative scheduling module, and the recomputation fragment optimization module to form a unified basis for memory constraints. The training convergence monitoring module dynamically adjusts the global strategy based on feedback from these modules.

Images