Adaptive priority scheduling method and device for large model training task

Abstract

The embodiment of the application provides a kind of self-adapting priority scheduling method and device of large model training task, through the performance monitoring system of innovatively designing, through feature analysis and state prediction, realize the effective planning of resource.Construct priority evaluation mechanism, combine multidimensional factors and decision making, establish reliable scheduling strategy.Introduce dynamic optimization, through execution monitoring and parameter adjustment, ensure the adaptability of scheduling.The method effectively solves the deficiencies of traditional technology in performance monitoring, priority evaluation and dynamic adjustment, etc., provides technical support for large model training.

Description

Adaptive Priority Scheduling Method and Apparatus for Large Model Training Tasks Technical Field

[0001] This application relates to the field of data processing, specifically to an adaptive priority scheduling method and apparatus for large model training tasks. Background Technology

[0002] Existing methods for scheduling large-scale model training tasks have significant shortcomings. Traditional systems perform poorly in performance monitoring and state prediction, failing to effectively grasp training characteristics and thus impacting scheduling efficiency.

[0003] Furthermore, existing technologies suffer from bottlenecks in prioritization and decision-making. Most systems lack robust benefit analysis mechanisms and migration cost assessment strategies, leading to inefficient resource allocation.

[0004] Existing systems have technical shortcomings in dynamic adjustment. The lack of in-depth analysis of execution states makes it difficult to achieve flexible task scheduling through policy optimization, thus impacting training performance. Solving these problems is crucial for improving the training capabilities of large models. Summary of the Invention

[0005] To address the problems in the existing technology, this application provides an adaptive priority scheduling method and apparatus for large model training tasks, which can effectively solve the shortcomings of traditional technologies in performance monitoring, priority evaluation and dynamic adjustment, and provide technical support for large model training.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] Firstly, this application provides an adaptive priority scheduling method for large model training tasks, including:

[0008] Collect training task operation index data, construct a performance feature matrix including computation step time and parameter synchronization delay, identify training stage and convergence state based on the performance feature matrix, generate task state vector including remaining workload, expected completion time and computing resource utilization, train prediction model according to task state vector, and output resource demand prediction curve within future time window.

[0009] The incremental training benefit is calculated based on the resource demand prediction curve. The task migration cost is estimated by combining the checkpoint size and network topology. A priority score is generated based on the task urgency, the incremental training benefit, the task migration cost, and the computing node affinity. A scheduling decision table containing resource allocation schemes, communication topology constraints, and batch dynamic adjustment strategies is constructed. The priority score is written into the scheduling decision table.

[0010] The scheduling decision table is sent to the resource manager for execution. The execution process is monitored to generate an execution log containing resource allocation status, task migration records, and training progress indicators. The parameters of the prediction model are adjusted according to the execution log. The scheduling decision table is updated under the constraints of checkpoint protection period and training stability, and task priority change information is recorded.

[0011] Furthermore, it also includes: obtaining data streams of computation step time, parameter synchronization delay, video memory usage, and device computation load from the training task runtime environment; performing time-series grouping and aggregation processing on the data streams; calculating the mean, variance, and trend slope of the indicators; constructing a performance indicator matrix containing multi-dimensional features; and extracting the number of training steps, the current batch size, and the number of model parameters to generate basic feature vectors.

[0012] The training phase discrimination marker is calculated based on the performance index matrix, the remaining training workload is calculated by combining the basic feature vector, the expected completion time is estimated according to the training progress curve, the computing device utilization rate is accumulated and averaged over a time window, and the phase discrimination marker, remaining workload, expected completion time, and accumulated average utilization rate are combined to generate a task status vector.

[0013] Furthermore, it also includes: constructing a training sample set containing a sequence of historical task state vectors and corresponding resource usage records; using a sliding time window to segment and label the training sample set; training a resource demand prediction model based on the gradient descent algorithm; and applying the resource demand prediction model to the current task state vector.

[0014] The original predicted values ​​output by the resource demand prediction model are smoothed to eliminate short-term fluctuations. The memory demand curve, computing load curve, and network bandwidth curve are generated by sampling at preset time intervals and then merged into a resource demand prediction curve.

[0015] Furthermore, it also includes: analyzing resource allocation schemes based on resource demand prediction curves, calculating training speed changes under different resource configurations, constructing a training benefit increment matrix by combining task completion deadlines and resource prices, and calculating task migration time costs and resource occupation costs based on checkpoint file size, inter-node network bandwidth, and data transmission paths.

[0016] The training benefit increment matrix is ​​mapped to the standard score range, the task migration time cost and resource consumption cost are normalized, and a weighted score vector is generated by combining the task urgency coefficient and the computing node affinity coefficient. The weighted score vector is then aggregated into a priority score scalar value.

[0017] Furthermore, it also includes: constructing a node resource allocation matrix based on the current computing cluster status, measuring bandwidth and detecting latency of communication links between computing nodes, generating a set of topology constraint rules based on communication performance indicators, determining the batch size adjustment range in combination with the computing characteristics of the training task, and constructing an initial decision table containing node identifiers, resource quotas, and link restrictions;

[0018] The priority score is mapped to a resource allocation weight. The resource quota value in the initial decision table is updated according to the resource allocation weight. The upper and lower limits of the batch size and the adjustment step size parameters are written to generate the final scheduling decision table.

[0019] Furthermore, it also includes: parsing the scheduling decision table into a set of scheduling instructions that the resource manager can execute, determining the task deployment order according to the node affinity rules, issuing resource allocation instructions according to the preset resource change step size, and starting the execution status collector to obtain computing node resource allocation data, task migration operation records, and training iteration counter values;

[0020] The resource allocation data of the computing nodes is sampled and aggregated, the migration success rate and time distribution in the task migration operation record are statistically analyzed, the training speed index is calculated based on the training iteration counter value, and the sampling results, statistical data and calculation index are written into the execution log.

[0021] Furthermore, it also includes: extracting time-series data of resource usage and actual training speed records from the execution log, calculating the deviation vector between the output value of the prediction model and the measured value, updating the parameters of the prediction model using an online learning method, and recalculating the predicted value of resource demand based on the updated prediction model;

[0022] The predicted resource demand is compared with the timestamp of the checkpoint to determine whether the protection period constraint is met. The training speed fluctuation amplitude is calculated to verify the training stability index. When the constraint is met, the resource configuration parameters in the scheduling decision table are updated, and a priority change record containing the adjustment time point, the reason for the change, and the scope of impact is generated.

[0023] Secondly, this application provides an adaptive priority scheduling device for large model training tasks, comprising:

[0024] The resource requirement determination module is used to collect training task operation index data, construct a performance feature matrix containing the computation step time and parameter synchronization delay, identify the training stage and convergence state based on the performance feature matrix, generate a task state vector containing the remaining workload, expected completion time and computational resource utilization, train a prediction model based on the task state vector, and output the resource requirement prediction curve within the future time window.

[0025] The resource scheduling decision module is used to calculate the training benefit increment based on the resource demand prediction curve, estimate the task migration cost by combining the checkpoint size and network topology, generate a priority score based on the task urgency, the training benefit increment, the task migration cost, and the computing node affinity, construct a scheduling decision table that includes resource allocation schemes, communication topology constraints, and batch dynamic adjustment strategies, and write the priority score into the scheduling decision table.

[0026] The priority scheduling execution module is used to send the scheduling decision table to the resource manager for execution, monitor the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators, adjust the prediction model parameters according to the execution log, update the scheduling decision table under the constraints of checkpoint protection period and training stability, and record task priority change information.

[0027] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the adaptive priority scheduling method for the large model training task.

[0028] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the adaptive priority scheduling method for the large model training task.

[0029] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the adaptive priority scheduling method for the large model training task.

[0030] As described above, this application provides an adaptive priority scheduling method and apparatus for large model training tasks. Through an innovative performance monitoring system, it achieves effective resource planning via feature analysis and state prediction. A priority evaluation mechanism is constructed, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively addresses the shortcomings of traditional techniques in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large model training. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a flowchart illustrating the adaptive priority scheduling method for large model training tasks in an embodiment of this application.

[0033] Figure 2 is a structural diagram of the adaptive priority scheduling device for large model training tasks in an embodiment of this application;

[0034] Figure 3 is a schematic diagram of the structure of the electronic device in the embodiment of this application.

[0035] Figure label:

[0036] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0039] To address the problems existing in current technologies, this application provides an adaptive priority scheduling method and apparatus for large model training tasks. Through an innovative performance monitoring system, it achieves effective resource planning via feature analysis and state prediction. A priority evaluation mechanism is constructed, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively solves the shortcomings of traditional techniques in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large model training.

[0040] To effectively address the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, and to provide technical support for large-scale model training, this application provides an embodiment of an adaptive priority scheduling method for large-scale model training tasks. Referring to Figure 1, the adaptive priority scheduling method for large-scale model training tasks specifically includes the following:

[0041] Step S101: Collect training task operation index data, construct a performance feature matrix containing the computation step time and parameter synchronization delay, identify the training stage and convergence state based on the performance feature matrix, generate a task state vector containing the remaining workload, expected completion time, and computational resource utilization, train a prediction model based on the task state vector, and output the resource demand prediction curve within the future time window.

[0042] This embodiment establishes an indicator collection channel between the training node and the management node to collect training task operation indicator data, specifically including computation step time, parameter synchronization latency, GPU memory usage, device computing load, and network bandwidth consumption. It also synchronously accesses request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns from the gateway side. Records from each source carry timestamps and task identifiers, are aligned to a fixed window using a unified clock, and are deduplicated by task and node. Fields are normalized in terms of units and value ranges according to a preset dictionary. Outliers are limited using the median deviation method; missing segments are not filled, only confidence weights are labeled for downstream calculation reduction.

[0043] A performance feature matrix is ​​constructed at the window level, with rows corresponding to time windows and columns containing at least the computation step time and parameter synchronization latency, and extended to include memory usage, device computational load, and bandwidth margin, to characterize the temporal pattern of the training phase. Based on this matrix, the moving average, variance, trend slope, and density of abrupt changes are calculated as inputs for phase identification. Phase identification employs a hybrid strategy of rules and lightweight models: a stable training phase is presumed when the computation step time converges and the parameter synchronization latency stabilizes; a checkpoint phase is presumed when periodic latency spikes occur accompanied by memory writes; and a warm-up and batching phase is classified as a significant decrease in step time and an increase in memory usage. Convergence status is determined jointly by the loss reduction rate and step size variance, outputting a phase discrimination label and convergence confidence.

[0044] A task state vector is generated based on stage discrimination. Training steps, current batch size, and model parameter count are used as basic features to estimate the remaining training workload. The training progress curve is adjusted by the cumulative effective steps and convergence confidence to obtain the expected completion time. Computational resource utilization is averaged cumulatively over a window, considering both device computational load and memory usage, and a confidence level is added. The stage discrimination marker, remaining workload, expected completion time, and cumulative average utilization are combined to form the task state vector, which is then bound to the task identifier and node group identifier before entering the prediction phase.

[0045] The resource demand forecasting model uses the Chinese name "Short-Term Demand Forecasting Model". The training samples consist of a sequence of historical task state vectors and their corresponding memory usage, computational load, and network throughput. Sliding window slicing is used, and stage markers are aligned. Gradient descent is employed for training to minimize the distance between predicted and measured values. During inference, the current task state vector is input, and the model outputs a sequence of memory requirements, computational load, and network bandwidth for several future windows. Exponential smoothing is used to suppress short-term spikes, and resource demand forecasting curves are generated by sampling at fixed intervals.

[0046] To facilitate subsequent scheduling calculations, this embodiment presents the coupling relationship between stability representation and demand constraints, which can be written as:

[0047] S = F(U; Π, Δ),

[0048] Where U is an indicator vector normalized from success rate, latency, business conversion, output distribution consistency, and resource utilization stability; Π is the set of weights for each indicator from the policy side; and Δ is the reduction factor for low-confidence windows and abnormal traffic hits. S does not directly participate in the stage identification in this step, but it is stored on the same timeline as the resource demand prediction curve for subsequent priority scoring and decision engine reading, limiting the scaling step size and migration trigger conditions within the resource constraints and stage protection period.

[0049] Step S102: Calculate the training benefit increment based on the resource demand prediction curve, estimate the task migration cost by combining the checkpoint size and network topology, generate a priority score based on the task urgency, the training benefit increment, the task migration cost, and the computing node affinity, construct a scheduling decision table that includes resource allocation scheme, communication topology constraints, and batch dynamic adjustment strategy, and write the priority score into the scheduling decision table.

[0050] This embodiment uses a resource demand prediction curve as input to construct a set of candidate resource configurations for multiple future windows, covering three adjustable dimensions: memory increment, computational concurrency, and network priority. For each candidate configuration, based on the predicted memory demand, computational load, and bandwidth usage, the training speed is simulated under that configuration. The training speed is represented by the number of effective steps per unit time, and the synchronization waiting time caused by changes in batch size is explicitly included. The reduction in training progress and resource usage cost are measured together, and the difference between the two after normalization yields the incremental training benefit. When the batch size increases and the synchronization latency exceeds the window threshold, the incremental benefit of the candidate is automatically reduced to avoid uneconomical expansion.

[0051] The migration cost is determined by both the checkpoint size and the network topology. The checkpoint size is derived from the byte size of the most recent persistent file, supplemented by the compression ratio and recovery overhead factor. The network topology is constructed as a transmission capacity matrix through bandwidth measurements and round-trip delay probing between node pairs. For possible migration paths, the opportunity cost of transmission time and bandwidth usage is estimated, and the loading and reconstruction time is added to form a combined cost of time and resource components. Tasks within the checkpoint protection period are not included in the migration candidate; tasks near the protection period boundary are penalized with a penalty coefficient to suppress high-risk handovers.

[0052] Task urgency is calculated based on remaining workload and expected completion time, and an urgency coefficient is given in conjunction with service level. Historical resource usage is used for fairness correction, ensuring that tasks with long-term high usage receive less benefit under the same urgency. Computation node affinity is estimated from past operational stability and data locality; nodes with high affinity are prioritized in expansion ranking, while nodes with low affinity have increased cost weight in migration path evaluation. Urgency, training benefit increment, migration cost, and computation node affinity are each interval normalized, linearly summed according to the weights set by the strategy parameters, and reduced by constraint factors for cases in the protection period, migration concurrency restrictions, or declining node health, resulting in a priority score scalar. This score lies in a closed interval between zero and one, with an additional confidence interval width as a buffer for subsequent threshold conditions.

[0053] Resource allocation schemes are generated based on priority scoring. Within the cluster's current available quota, allocation is discretized, specifying the incremental GPU memory, concurrent computing power limit, and network queue priority for each task. For tasks requiring cross-node expansion, communication topology constraints are extracted using the transmission capacity matrix, marking insurmountable low-bandwidth links and highly congested nodes to restrict parameter synchronization path selection. A dynamic batch adjustment strategy is implemented around the upper and lower limits and step size of the batch size, utilizing peak and valley periods given by the resource demand forecast curve to schedule batch expansion and contraction, and setting cooldown times to avoid frequent jitter.

[0054] The scheduling decision table, indexed by task identifiers, records resource allocation schemes, communication topology constraints, batch dynamic adjustment strategies, and priority scores for each item. Each record carries the generation time, applicable window, and constraint fingerprint, used by the execution layer to verify change step sizes and concurrency limits. The resource demand prediction curve generated in previous steps is used in this step to calculate training benefit increments and screen feasible configurations; the priority score obtained in this step will participate in action ranking and conflict resolution in subsequent decision engines, and will be compared with actual benefits after execution log recollection, serving as the basis for weight calibration and threshold condition revision in the next cycle.

[0055] Step S103: Send the scheduling decision table to the resource manager for execution, monitor the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators, adjust the prediction model parameters according to the execution log, update the scheduling decision table while satisfying the checkpoint protection period and training stability constraints, and record task priority change information.

[0056] This embodiment enters the execution phase after generating the scheduling decision table. The management node parses the scheduling decision table into an executable instruction set, determines the distribution order based on task priority scoring, and adjusts the memory quota, concurrent computing power limit, and network queue priority within each safety window using discrete step sizes. For entries containing communication topology constraints, the link bandwidth and latency are first verified at the target node to ensure they meet threshold conditions before triggering scaling up / down or migration. If the checkpoint protection period has not yet expired, only in-situ scaling up or batch scaling down is performed. Migration actions are bound to rollback points, recording the source node and target node, checkpoint path, and estimated recovery time to prevent training rollback caused by abnormal interruptions.

[0057] The execution process is continuously monitored by a status collector, which collects resource allocation status, task migration records, and training progress metrics. Resource allocation status includes memory usage curves, device computational load, and network usage, aggregated within a fixed window and compared with the instruction target value. Task migration records include trigger time, transmission time, recovery time, and success flags, indicating whether the protection period boundary has been hit. Training progress metrics are calculated from the iteration counter and the number of effective steps per unit time, along with batch size and synchronization wait time, to explain progress fluctuations. All the above data is written to the execution log by task and node identifiers, with added confidence weights and window fingerprints for subsequent verification.

[0058] Online calibration of the short-term demand forecasting model is performed based on execution logs. Resource usage time series and measured training speed are read, and the deviation vector between predicted and measured values ​​is calculated. Model parameters are adjusted using a gradient update method with small steps. If the deviation maintains a stable direction within a continuous window, the learning step size is increased; otherwise, a cooldown period is entered to pause updates. The updated forecasting model re-infers the memory, computation, and bandwidth requirements for future windows based on the current task state vector, generating a new resource demand forecast curve, which is then aligned with the latest stage judgment results to avoid misjudgments caused by stage changes.

[0059] Update the scheduling decision table under the constraints of checkpoint protection period and training stability.

[0060] First, the timestamp of the most recent checkpoint write is read to determine if migration and batch expansion have triggered the protection period. Then, the training speed fluctuation and error event count are calculated. If the fluctuation is low and the error count does not exceed the threshold, the memory and concurrency limits are allowed to be relaxed; otherwise, only batch reduction or network priority adjustment is allowed. Changes that meet the constraints are written to a new version of the scheduling decision table, and an effective window and cooldown time are set; changes that do not meet the constraints are turned into pending entries, retaining the reasons and triggering indicators.

[0061] Each table version switch generates a task priority change record, including the change time, a summary of the triggering reason, the set of tasks affected, the extent of resource adjustments, and a list of involved nodes and links. This record is cross-indexed with migration records and training progress metrics in the execution log, forming a traceable decision-making loop. In subsequent cycles, the decision engine uses the latest prediction model, execution log, and priority change record as joint inputs to select the next executable action, ensuring consistency between resource adjustments and training stability constraints.

[0062] As described above, the adaptive priority scheduling method for large model training tasks provided in this application can achieve effective resource planning through innovative performance monitoring system design, feature analysis, and state prediction. It constructs a priority evaluation mechanism, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively solves the shortcomings of traditional techniques in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large model training.

[0063] In one embodiment of the adaptive priority scheduling method for large model training tasks in this application, the method may further include the following:

[0064] Step S201: Obtain the computation step time, parameter synchronization delay, memory usage, and device computation load data stream from the training task runtime environment; perform time-series grouping and aggregation processing on the data stream; calculate the mean, variance, and trend slope of the indicators; construct a performance indicator matrix containing multi-dimensional features; and extract the number of training steps, the current batch size, and the number of model parameters to generate basic feature vectors.

[0065] Step S202: Calculate the training phase discrimination label based on the performance index matrix, calculate the remaining training workload by combining the basic feature vector, estimate the expected completion time according to the training progress curve, accumulate and average the computing device utilization rate over a time window, and combine the phase discrimination label, remaining workload, expected completion time, and accumulated average utilization rate to generate a task status vector.

[0066] In this embodiment, a lightweight data acquisition device is deployed on the training node to continuously acquire data streams from the training task runtime environment, including computation step time, parameter synchronization delay, video memory usage, and device computation load.

[0067] Each record carries a timestamp, task identifier, node identifier, and batch identifier. To eliminate clock deviations from different sources, a unified mapping to the baseline clock of the management node is performed, and alignment is carried out according to a fixed window.

[0068] After alignment, time-series grouping and aggregation are performed using tasks and nodes as keys. For each window, the mean, variance, and least-squares-based trend slope of the aforementioned metrics are calculated, while the missing test ratio and outlier markers are recorded for downstream reduction. This constructs a performance metric matrix, with rows corresponding to time windows and columns covering computation step time, parameter synchronization latency, GPU memory usage, device computational load, and their statistics. Furthermore, the number of training steps, current batch size, and model parameter count are extracted from the runtime context, generating basic feature vectors by task dimension, which are then aligned with the performance metric matrix on the time axis.

[0069] The training phase is determined based on a performance metric matrix. The joint pattern of step length, parameter synchronization latency, and memory usage is used as the criterion: stable training occurs when the step length is stable and synchronization latency fluctuations converge; a periodic increase in latency accompanied by a peak in memory writes indicates a checkpoint phase; a rapid decrease in step length and an increase in memory usage indicate a warm-up and batch expansion phase; and synchronization blocking occurs when the device has a high computational load and increasing latency. Phase determination is output with phase markers and confidence levels at the window level. Combining the training steps, current batch size, and model parameter count in the basic feature vector, the remaining training workload is calculated according to a pre-defined milestone table. The training progress curve is normalized to the target steps using the cumulative effective steps, and the stage confidence level corrects for abrupt error changes, thereby estimating the expected completion time.

[0070] To accurately reflect the supply-side status, the computational load and memory usage of the equipment are cumulatively averaged over a window on the time axis of each task, resulting in a smoothed index of computing equipment utilization. Finally, the stage discrimination marker, remaining training workload, expected completion time, and cumulative average utilization are combined to form a task state vector, accompanied by a window timestamp, task identifier, and node group identifier. This task state vector is input to the subsequent short-term demand forecasting model, and its output resource demand forecast curve will be referenced in priority scoring and resource adjustment calculations. Simultaneously, the stage discrimination marker is used to limit the triggering conditions for migration and batch expansion, ensuring that no high-risk actions occur during checkpoint protection periods and synchronization blocking periods.

[0071] In one embodiment of the adaptive priority scheduling method for large model training tasks in this application, the method may further include the following:

[0072] Step S301: Construct a training sample set containing a sequence of historical task state vectors and corresponding resource usage records. Use a sliding time window to segment and label the training sample set. Train a resource demand prediction model based on the gradient descent algorithm. Apply the resource demand prediction model to the current task state vector.

[0073] Step S302: Smooth the original predicted values ​​output by the resource demand prediction model to eliminate short-term fluctuations, and sample and generate memory demand curve, calculation load curve and network bandwidth curve according to a preset time interval, and merge the three curves into a resource demand prediction curve.

[0074] This embodiment assumes an endogenous data closure loop and uses the task state vector output from upstream steps as input to construct a training sample set for resource demand prediction. The collection scope covers a continuous window on the historical timeline. Each sample consists of a sequence of task state vectors and the corresponding memory usage, device computing load, and network throughput within the window. To ensure the time consistency between the labels and the input, all records are aligned to a unified clock and deduplicated by a two-level key (task and node). Missing segments are only labeled with low-confidence weights without numerical padding. Samples are labeled in segments according to sliding time windows. The window length and step size are adaptively set based on the stage discrimination results. A longer window is used when the stage is stable, and the window is shortened when the stage switches to retain transition information.

[0075] The resource demand forecasting model is named the "Short-Term Demand Forecasting Model" in Chinese. The input is a fixed-length sequence of task state vectors, and the output is a sequence of memory requirements, computational load, and network bandwidth for several future windows. During training, a gradient descent algorithm is used to minimize the weighted loss, which consists of three parts: the distance between the predicted and measured values, the smoothing constraint of adjacent prediction windows, and the continuity penalty at stage boundaries. Weights are related to window confidence and stage confidence; high-confidence samples have higher weights on the target term, and stage transition segments have lower weights on the smoothing term, avoiding over-smoothing of real transitions. Training uses mini-batch iterations with an early stopping condition; parameters are frozen when the validation loss no longer decreases within a certain number of rounds. After model convergence, the current task state vector is read, and the three types of original predicted values ​​for the future windows are output, while retaining the confidence level label at each time point.

[0076] The original predicted values ​​are affected by instantaneous disturbances, and directly using them for decision-making can easily lead to unnecessary resource fluctuations. This embodiment performs smoothing on the three types of channels separately, using a combination of exponential weighting and piecewise linearity: during the stable training phase, the time decay window is increased to emphasize the trend; during the checkpoint phase, the decay window is reduced and short-term spikes are allowed to be retained. An amplitude threshold is set for non-physical jumps across the window; portions exceeding the threshold are replaced by linear interpolations on either side. After smoothing, synchronous sampling is performed at preset time intervals, consistent with the resource manager's change-effective window, resulting in memory demand curves, computational load curves, and network bandwidth curves. Each sampling point includes a predicted value and a confidence level.

[0077] To ensure consistent subsequent referencing, the three curves are aligned and merged according to their time indices to form a resource demand prediction curve. Structurally, this curve uses time as the primary key, with fields including memory requirements, computational load, network bandwidth, and confidence level. This curve serves as the primary input for subsequent computational training efficiency increments and for selecting executable resource configurations. Simultaneously, it acts as a baseline during the execution log retrieval phase, measuring prediction deviations and triggering online calibration of the short-term demand prediction model.

[0078] In real-world scenarios, if a training task has just completed a checkpoint write, there may be interpretable uplinks in network throughput and parameter synchronization latency within a short period. The smoothing and sampling strategies described above prevent such periodic peak errors from spreading into longer future windows, thus avoiding excessive cross-node migration instructions at the scheduling layer. Therefore, the resource demand prediction curve retains both the necessary short-term response and the stability available for subsequent scheduling calculations, ensuring temporal consistency with priority scoring and scheduling decision tables.

[0079] In one embodiment of the adaptive priority scheduling method for large model training tasks in this application, the method may further include the following:

[0080] Step S401: Analyze the resource allocation scheme based on the resource demand prediction curve, calculate the training speed change under different resource configurations, construct the training benefit increment matrix by combining the task completion deadline and resource price, and calculate the task migration time cost and resource occupation cost based on the checkpoint file size, network bandwidth between nodes, and data transmission path.

[0081] Step S402: Map the training benefit increment matrix to the standard score range, normalize the task migration time cost and resource consumption cost, generate a weighted score vector by combining the task urgency coefficient and the computing node affinity coefficient, and aggregate the weighted score vector into a priority score scalar value.

[0082] This embodiment uses an intrinsic resource demand prediction curve as input to jointly interpret the memory, computing load, and network bandwidth for multiple future windows, generating a set of executable resource allocation candidate schemes. Each candidate consists of three components: memory increment, concurrency limit, and communication priority, with constraints not exceeding the minimum granularity and the current available limit. Training speed is represented by the effective number of steps per unit time, establishing a causal relationship between batch size, synchronization latency, and bandwidth usage: when memory and concurrency are increased simultaneously, if synchronization latency does not increase, the speed gain is effective; if bandwidth becomes a bottleneck, the gain is reduced according to the bottleneck ratio. Based on this, the training speed change for all candidates within each time window is calculated, and a training benefit increment matrix is ​​constructed by combining the task completion deadline and resource price; the matrix elements represent the progress reduction minus the resource cost under a given window and configuration, serving as the core input for subsequent scoring.

[0083] The migration-related costs are estimated based on the checkpoint file size and network topology. The checkpoint file size directly indicates the amount of data transmitted, while the network topology forms a transmission capacity matrix through inter-node bandwidth and round-trip latency, and indicates the link congestion level. For each candidate migration path, time cost and resource consumption cost are calculated. The time cost includes transmission time and loading / reconstruction time, while the resource consumption cost reflects the impact of bandwidth usage during migration on tasks within the same domain. Tasks within the checkpoint protection period do not participate in migration evaluation; tasks near the boundary have a penalty coefficient added to their cost to reduce the probability of triggering migration actions during unstable periods.

[0084] To achieve comparable scoring, this embodiment maps the training benefit increment matrix to a fixed score range and normalizes the task migration time cost and resource consumption cost, obtaining three types of quantities with the same dimensions. A task urgency coefficient and a computation node affinity coefficient are constructed. Urgency is given by the remaining workload and expected completion time, combined with service level to form a weighted coefficient; affinity is calculated based on past stable operation on the node and data locality, with higher values ​​indicating a greater tendency to expand on the original node. The four factors are combined into a weighted scoring vector, aggregated using weights given by the strategy parameters, to obtain a priority score scalar value. The score lies in a closed interval between zero and one, with an accompanying confidence interval width for subsequent threshold condition buffering and conflict resolution.

[0085] In a typical scenario, if the resource demand prediction curve shows peak computational load in the next two windows while network bandwidth remains abundant, then candidates for increasing concurrency and adding GPU memory will receive positive values ​​in the training benefit increment matrix. If, simultaneously, a checkpoint has just been completed and link congestion between the target node and the source node is low, the normalized values ​​of migration time cost and resource occupancy cost are small, resulting in weak suppression of the overall score, and the task urgency is high, then the aggregated priority score will be higher. This score is written back to the scheduling decision table and, together with the resource allocation scheme, communication topology constraints, and batch dynamic adjustment strategy, is distributed to the execution layer. In subsequent steps, the execution log will verify the actual benefits according to this score, which will be used for weight revision and threshold update in the next cycle.

[0086] In one embodiment of the adaptive priority scheduling method for large model training tasks in this application, the method may further include the following:

[0087] Step S501: Construct a node resource allocation matrix based on the current computing cluster status, measure bandwidth and detect latency of communication links between computing nodes, generate a set of topology constraint rules based on communication performance indicators, determine the batch size adjustment range in combination with the computing characteristics of the training task, and construct an initial decision table containing node identifiers, resource quotas, and link restrictions.

[0088] Step S502: Map the priority score to a resource allocation weight, update the resource quota value in the initial decision table according to the resource allocation weight, write the upper and lower limits of the batch size and the adjustment step size parameters, and generate the final scheduling decision table.

[0089] This embodiment constructs a node resource allocation matrix based on the real-time status of the cluster. The inputs are the available GPU memory, computational utilization, power consumption limit, and container concurrency usage for each node, all timestamped and identified by the node. These are aligned to a fixed window using a unified clock, and duplicate samples across windows are removed. For each node, the robust mean and fluctuation range of available GPU memory, available computing power, and scalable container concurrency within the window are statistically analyzed to form node-side resource elements. To characterize communication conditions, bandwidth measurement and latency probing are performed between all node pairs to obtain the effective throughput and round-trip latency of the links. Congestion markers are then overlaid to generate a set of communication performance indicators. This constructs a node resource allocation matrix, with rows corresponding to nodes and columns containing summaries of available GPU memory, available computing power, scalable concurrency, and bandwidth and latency from links to adjacent nodes.

[0090] A set of topology constraint rules is generated based on communication performance indicators. Threshold conditions are set using link capacity, latency, and congestion markers to label low-bandwidth links, long-latency cross-rack links, and temporarily disabled links, forming three categories of rules: untraversable, degradeable, and priority paths. The computational characteristics of the training task are analyzed to calculate the single-step computing power requirement, memory baseline, and parameter synchronization ratio. Peak and valley positions within the future window are identified by combining the resource demand prediction curve. This determines the batch size adjustment range, giving the upper and lower bounds of the batch size and the step size range, requiring that the adjustment does not introduce instantaneous memory overflow or a surge in synchronization waiting.

[0091] An initial decision table is constructed based on the above elements. Each task record in the table includes a set of candidate node identifiers, an initial resource quota calculated per node, and available link limits. The initial resource quota is obtained by matching the node resource allocation matrix with the task calculation characteristics, adhering to the minimum granularity and the current available upper limit, and does not cross topological constraints. The batch size adjustment range and link limits are also written in, indicating the effective window and cooldown time. This initial decision table will serve as the carrier for weight mapping and resource refinement updates.

[0092] Priority scores are provided by upstream steps S401 and S402. In this embodiment, these scores are mapped to resource allocation weights. The mapping function satisfies monotonicity and sets a safety net factor to prevent low-scoring tasks from being completely squeezed out. Normalization is performed along the task dimension to obtain a cross-task weight vector. Based on the resource allocation weights, the resource quota values ​​in the initial decision table are updated proportionally. The memory increment, concurrency limit, and network queue priority are jointly determined according to their channel weights and the available margins of nodes. If a node cannot meet the allocation increase under communication constraints, it falls back to the set of nodes with the second-best weights to maintain a balanced total resource budget.

[0093] Batch dynamic parameters are finalized at this stage. The upper and lower limits of the batch size and the adjustment step size parameters are written based on the batch size adjustment range. The step size is subject to three constraints: the memory safety margin threshold, the predicted synchronization wait rate threshold, and the cooldown period length. When the prediction curve indicates an approaching load peak, the step size automatically converges to a smaller value and the uplink direction is restricted; batch expansion is allowed during load troughs, but disabled links in the topology constraints must not be triggered.

[0094] The final scheduling decision table includes task identifiers, node resource quotas, link limits, and batch dynamic parameters, and records a weight snapshot mapped from priority scores for execution layer verification. This table is aligned with the resource manager's change application window, and subsequent steps parse this table into discrete adjustment instructions during the distribution process. The execution log will verify whether resource quotas take effect according to weights and whether batch adjustments follow cooldown times and link constraints. The verification results will be fed back to the weight mapping and batch step size strategy, forming a closed-loop correction for the next cycle.

[0095] In one embodiment of the adaptive priority scheduling method for large model training tasks in this application, the method may further include the following:

[0096] Step S601: Parse the scheduling decision table into a set of scheduling instructions that the resource manager can execute, determine the task deployment order according to the node affinity rules, issue resource allocation instructions according to the preset resource change step size, and start the execution status collector to obtain computing node resource allocation data, task migration operation records, and training iteration counter values.

[0097] Step S602: Sample and aggregate the computing node resource allocation data, statistically analyze the migration success rate and time distribution in the task migration operation record, calculate the training speed index based on the training iteration counter value, and write the sampling results, statistical data, and calculation index into the execution log.

[0098] In this embodiment, the scheduling decision table generated in the previous stage is used as input and parsed into a set of scheduling instructions that can be executed by the resource manager.

[0099] Each instruction includes a task identifier, target node, memory increment, concurrency limit, network priority, and whether it involves migration and its rollback point. Task deployment order is determined based on node affinity rules, with affinity derived from historical stable operation records and data locality. Nodes with high affinity are prioritized for expansion, while nodes with low affinity but high idle time standby in the reserve sequence. Instruction issuance employs an offline step-by-step strategy, with step size bound to resource channels. Memory and concurrency are distributed across separate channels, and the current node capacity and topology constraints are verified within each safety window. Instructions involving migration are only registered during the checkpoint protection period and do not trigger actual migration.

[0100] To ensure closed-loop observation, the execution status acquisition device is started.

[0101] The data collector pulls resource allocation data from the node side, including fields such as GPU memory usage, device computing load, and network usage, and aggregates them into a time series using a fixed window. On the task side, it records the trigger time of migration operations, source and target nodes, transmission time, and recovery time, distinguishing between success and failure and attaching exception codes. On the training side, it reads the iteration counter value to calculate the effective steps per unit time and the batch size of the nearest window to explain progress changes. All data entries are appended with task and node identifiers and aligned with a unified clock. Missing test segments are assigned only low confidence weights and are not interpolated.

[0102] Resource allocation data for computing nodes is sampled and aggregated to form a summary metric consistent with the distribution window, including the mean channel utilization rate, fluctuation range, and deviation from the target value of the instruction. Migration operation records are statistically analyzed by task and node pairs to obtain the migration success rate and time distribution. The time distribution is expressed in quantiles, and it is marked whether congested links are crossed and whether the protection period boundary is touched. The training speed metric is numerically calculated based on an iterative counter. The calculation method is the ratio of the effective steps within the window to the window duration. The speed is interpreted and corrected by the batch size variation to avoid the instantaneous jitter caused by batch expansion being misinterpreted as degradation.

[0103] This embodiment writes sampling results, statistical data, and calculated metrics into the execution log. The execution log uses the window as the primary key, along with the scheduling decision table version number and effective time, facilitating accurate mapping to specific instruction sets during subsequent replay. To prepare for subsequent online calibration, each record is marked with a confidence level and an anomaly flag. When resource quotas fail to take effect according to the step size or migration failures are concentrated on a specific link, the corresponding window is marked as low confidence. The execution log is read in the next cycle for two purposes:

[0104] First, compare the short-term demand forecasting model parameters with the resource demand forecasting curve;

[0105] Second, the priority scoring weights and threshold conditions are revised by comparing them with the estimated values ​​of training benefit increment and transfer cost.

[0106] Through the aforementioned data links, a closed loop is formed from decision-making to observation feedback, ensuring that the pace of resource changes is consistent with the training stability constraints.

[0107] In one embodiment of the adaptive priority scheduling method for large model training tasks in this application, the method may further include the following:

[0108] Step S701: Extract time-series data of resource usage and actual training speed records from the execution log, calculate the deviation vector between the output value of the prediction model and the measured value, update the parameters of the prediction model using an online learning method, and recalculate the predicted value of resource demand based on the updated prediction model.

[0109] Step S702: Compare the predicted resource demand value with the timestamp of the checkpoint to determine whether the protection period constraint is met, calculate the training speed fluctuation amplitude to verify the training stability index, update the resource configuration parameters in the scheduling decision table when the constraint is met, and generate a priority change record containing the adjustment time point, the reason for the change, and the scope of impact.

[0110] This embodiment uses execution logs as input, reading time-series data on resource usage and actual training speed records for each task. Data is aligned to a window granularity, with fields including GPU memory usage, device computational load, network throughput, and iteration counter values. Missing test segments are simply marked with low confidence and no interpolation is performed. To avoid bias in the learning results caused by abnormal windows, a sample weight vector is constructed, with weights adjusted according to log confidence and stage confidence. The previous output of the resource demand prediction model is paired with the measured values ​​within the same window, and a bias vector is calculated, splitting it into three parts: a GPU memory channel, a computational channel, and a bandwidth channel to distinguish the source of error. The error is then denoised before entering the parameter update stage.

[0111] The prediction model is named "Short-Term Demand Prediction Model" in Chinese. Online learning primarily uses small-step gradient updates. The objective function consists of a weighted residual term and a smoothing regularization term. The residual term mitigates systematic biases, while the smoothing term suppresses overfitting to instantaneous spikes. The update strategy includes a cooldown period and boundary protection: when the error directions of consecutive windows are inconsistent, updates are paused and resumed only when convergence signs appear; when a stage switches, only the subset of parameters relevant to the new stage is updated. After each parameter update, the memory requirements, computational load, and network bandwidth for several future windows are re-inferred based on the latest task state vector, forming new resource demand predictions, and the confidence level at each time point is output.

[0112] To ensure scheduling safety, this embodiment compares the predicted resource demand with the timestamps of the checkpoints to determine whether the protection period is met.

[0113] Migration is prohibited during the protection period, but in-situ scaling is allowed. A buffer zone is set at the end of the protection period, with an additional reduction factor applied to migration-related requests to prevent excessive actions triggered at the boundary. Simultaneously, the training speed fluctuation is calculated, expressed as the ratio of effective steps to time within a window, and then explained and corrected for by batch size changes, distinguishing between short-term fluctuations caused by batch expansion and true degradation. Stability metrics are jointly determined by the fluctuation threshold and the error event count; resource allocation can only be increased or the concurrency limit raised when both are satisfied.

[0114] When the constraints are met, the scheduling decision table is updated in small steps based on the new resource demand forecasts. The update scope includes memory increments, concurrency limits, and network priorities, all of which are limited by the minimum granularity and total quota boundaries; entries involving cross-node actions need to be re-verified for topology constraints and node affinity. Each change also generates a priority change record, which includes the adjustment time, a summary of the indicators for the reason for the change, the set of affected tasks and the set of involved nodes, and the corresponding scheduling decision table version number for easy backtracking.

[0115] In a common scenario, execution logs show that the predicted computation channel for a certain task is low while stability metrics are met. The short-term demand prediction model, through online learning, improves its sensitivity to peak periods and predicts higher concurrency demands in the next cycle. Based on this, the system increases the concurrency limit within the non-protection window and allows for a cooldown period. If subsequent logs show that the speed does not improve as expected, the task's weight is reduced and the concurrency configuration is rolled back in the next calibration round. Priority change records are updated synchronously, forming a closed loop from error assessment to configuration implementation.

[0116] To effectively address the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, and to provide technical support for large model training, this application provides an embodiment of an adaptive priority scheduling device for large model training tasks, which implements all or part of the adaptive priority scheduling method for the aforementioned large model training task. Referring to Figure 2, the adaptive priority scheduling device for large model training tasks specifically includes the following components:

[0117] The resource requirement determination module 10 is used to collect training task operation index data, construct a performance feature matrix including the computation step time and parameter synchronization delay, identify the training stage and convergence state based on the performance feature matrix, generate a task state vector including remaining workload, expected completion time and computational resource utilization, train a prediction model based on the task state vector, and output the resource requirement prediction curve within the future time window.

[0118] The resource scheduling decision module 20 is used to calculate the training benefit increment based on the resource demand prediction curve, estimate the task migration cost by combining the checkpoint size and network topology, generate a priority score based on the task urgency, the training benefit increment, the task migration cost, and the computing node affinity, construct a scheduling decision table that includes resource allocation scheme, communication topology constraints, and batch dynamic adjustment strategy, and write the priority score into the scheduling decision table.

[0119] The priority scheduling execution module 30 is used to send the scheduling decision table to the resource manager for execution, monitor the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators, adjust the prediction model parameters according to the execution log, update the scheduling decision table under the constraints of checkpoint protection period and training stability, and record task priority change information.

[0120] As described above, the adaptive priority scheduling device for large model training tasks provided in this application can achieve effective resource planning through innovative performance monitoring system design, feature analysis, and state prediction. It constructs a priority evaluation mechanism, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively solves the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large model training.

[0121] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, and to provide technical support for large model training, this application provides an embodiment of an electronic device for implementing all or part of the adaptive priority scheduling method for the large model training task. The electronic device specifically includes the following components:

[0122] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to transmit information between the adaptive priority scheduling device for large model training tasks and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the adaptive priority scheduling method for large model training tasks and the embodiments of the adaptive priority scheduling device for large model training tasks in the present embodiment, the contents of which are incorporated herein, and repeated details will not be described again.

[0123] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0124] In practical applications, the adaptive priority scheduling method for large model training tasks can be partially executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0125] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0126] Figure 3 is a schematic block diagram of the system configuration of an electronic device 9600 according to an embodiment of this application. As shown in Figure 3, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0127] In one embodiment, the adaptive priority scheduling method for large model training tasks can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0128] Step S101: Collect training task operation index data, construct a performance feature matrix containing the computation step time and parameter synchronization delay, identify the training stage and convergence state based on the performance feature matrix, generate a task state vector containing the remaining workload, expected completion time, and computational resource utilization, train a prediction model based on the task state vector, and output the resource demand prediction curve within the future time window.

[0129] Step S102: Calculate the training benefit increment based on the resource demand prediction curve, estimate the task migration cost by combining the checkpoint size and network topology, generate a priority score based on the task urgency, the training benefit increment, the task migration cost, and the computing node affinity, construct a scheduling decision table that includes resource allocation scheme, communication topology constraints, and batch dynamic adjustment strategy, and write the priority score into the scheduling decision table.

[0130] Step S103: Send the scheduling decision table to the resource manager for execution, monitor the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators, adjust the prediction model parameters according to the execution log, update the scheduling decision table while satisfying the checkpoint protection period and training stability constraints, and record task priority change information.

[0131] As described above, the electronic device provided in this application, through an innovative performance monitoring system design, achieves effective resource planning through feature analysis and state prediction. It constructs a priority evaluation mechanism, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively solves the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large-scale model training.

[0132] In another embodiment, the adaptive priority scheduling device for large model training tasks can be configured separately from the central processing unit 9100. For example, the adaptive priority scheduling device for large model training tasks can be configured as a chip connected to the central processing unit 9100, and the adaptive priority scheduling method function for large model training tasks can be implemented through the control of the central processing unit.

[0133] As shown in Figure 3, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily include all the components shown in Figure 3; furthermore, the electronic device 9600 may also include components not shown in Figure 3, as can be found in existing technologies.

[0134] As shown in Figure 3, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0135] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0136] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0137] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0138] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0139] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0140] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0141] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the adaptive priority scheduling method for large model training tasks with server or client execution subjects in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the adaptive priority scheduling method for large model training tasks with server or client execution subjects in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0142] Step S101: Collect training task operation index data, construct a performance feature matrix containing the computation step time and parameter synchronization delay, identify the training stage and convergence state based on the performance feature matrix, generate a task state vector containing the remaining workload, expected completion time, and computational resource utilization, train a prediction model based on the task state vector, and output the resource demand prediction curve within the future time window.

[0143] Step S102: Calculate the training benefit increment based on the resource demand prediction curve, estimate the task migration cost by combining the checkpoint size and network topology, generate a priority score based on the task urgency, the training benefit increment, the task migration cost, and the computing node affinity, construct a scheduling decision table that includes resource allocation scheme, communication topology constraints, and batch dynamic adjustment strategy, and write the priority score into the scheduling decision table.

[0144] Step S103: Send the scheduling decision table to the resource manager for execution, monitor the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators, adjust the prediction model parameters according to the execution log, update the scheduling decision table while satisfying the checkpoint protection period and training stability constraints, and record task priority change information.

[0145] As described above, the computer-readable storage medium provided in this application, through an innovative performance monitoring system design, achieves effective resource planning through feature analysis and state prediction. It constructs a priority evaluation mechanism, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively solves the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large-scale model training.

[0146] Embodiments of this application also provide a computer program product capable of implementing all steps of the adaptive priority scheduling method for large model training tasks, where the execution subject is a server or client, as described in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the adaptive priority scheduling method for large model training tasks. For example, the computer program / instruction implements the following steps:

[0147] Step S101: Collect training task operation index data, construct a performance feature matrix containing the computation step time and parameter synchronization delay, identify the training stage and convergence state based on the performance feature matrix, generate a task state vector containing the remaining workload, expected completion time, and computational resource utilization, train a prediction model based on the task state vector, and output the resource demand prediction curve within the future time window.

[0148] Step S102: Calculate the training benefit increment based on the resource demand prediction curve, estimate the task migration cost by combining the checkpoint size and network topology, generate a priority score based on the task urgency, the training benefit increment, the task migration cost, and the computing node affinity, construct a scheduling decision table that includes resource allocation scheme, communication topology constraints, and batch dynamic adjustment strategy, and write the priority score into the scheduling decision table.

[0149] Step S103: Send the scheduling decision table to the resource manager for execution, monitor the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators, adjust the prediction model parameters according to the execution log, update the scheduling decision table while satisfying the checkpoint protection period and training stability constraints, and record task priority change information.

[0150] As described above, the computer program product provided in this application, through an innovative performance monitoring system design, achieves effective resource planning through feature analysis and state prediction. It constructs a priority evaluation mechanism, combining multi-dimensional factors and decision-making to establish a reliable scheduling strategy. Dynamic optimization is introduced, ensuring scheduling adaptability through execution monitoring and parameter adjustment. This method effectively solves the shortcomings of traditional technologies in performance monitoring, priority evaluation, and dynamic adjustment, providing technical support for large-scale model training.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0155] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An adaptive priority scheduling method for large model training tasks, characterized in that, The method includes: collecting training task operation index data, constructing a performance feature matrix containing computation step time and parameter synchronization delay, identifying training stages and convergence states based on the performance feature matrix, generating a task state vector containing remaining workload, expected completion time, and computing resource utilization, training a prediction model based on the task state vector, and outputting a resource demand prediction curve within a future time window. This includes: obtaining data streams of computation step time, parameter synchronization delay, GPU memory usage, and device computing load from the training task operation environment; performing time-series grouping and aggregation processing on the data streams; calculating the index mean, variance, and trend slope; constructing a performance index matrix containing multi-dimensional features; extracting training steps, current batch size, and model parameter count to generate basic feature vectors; calculating training stage discrimination markers based on the performance index matrix; calculating the remaining training workload in conjunction with the basic feature vectors; estimating the expected completion time based on the training progress curve; accumulating and averaging the computing device utilization over a time window; combining the stage discrimination markers, remaining workload, expected completion time, and accumulated average utilization to generate a task state vector; and constructing training samples containing historical task state vector sequences and corresponding resource usage records. The training sample set is segmented and labeled using a sliding time window. A resource demand prediction model is trained based on the gradient descent algorithm and applied to the current task state vector. The original predicted values ​​output by the resource demand prediction model are smoothed to eliminate short-term fluctuations. Memory demand curves, computational load curves, and network bandwidth curves are generated by sampling at preset time intervals and merged into a resource demand prediction curve. The training benefit increment is calculated based on the resource demand prediction curve. The task migration cost is estimated by combining the checkpoint size and network topology. A priority score is generated based on the task urgency, the training benefit increment, the task migration cost, and the computation node affinity. A scheduling decision table containing resource allocation schemes, communication topology constraints, and batch dynamic adjustment strategies is constructed, and the priority score is written into the scheduling decision table. The scheduling decision table is sent to the resource manager for execution. The execution process is monitored to generate an execution log containing resource allocation status, task migration records, and training progress indicators. The prediction model parameters are adjusted based on the execution log. The scheduling decision table is updated under the constraints of checkpoint protection period and training stability, and task priority change information is recorded.

2. The adaptive priority scheduling method for large model training tasks according to claim 1, characterized in that, The process of calculating the training benefit increment based on the resource demand prediction curve, estimating the task migration cost by combining checkpoint size and network topology, and generating a priority score based on task urgency, the training benefit increment, the task migration cost, and computing node affinity includes: analyzing resource allocation schemes based on the resource demand prediction curve, calculating training speed changes under different resource configurations, constructing a training benefit increment matrix by combining task completion deadline and resource price, calculating task migration time cost and resource occupation cost based on checkpoint file size, inter-node network bandwidth, and data transmission path; mapping the training benefit increment matrix to a standard score range, normalizing the task migration time cost and resource occupation cost, generating a weighted score vector by combining task urgency coefficient and computing node affinity coefficient, and aggregating the weighted score vector into a priority score scalar value.

3. The adaptive priority scheduling method for large model training tasks according to claim 1, characterized in that, The construction of a scheduling decision table, which includes resource allocation schemes, communication topology constraints, and batch dynamic adjustment strategies, and the writing of the priority score into the scheduling decision table, includes: constructing a node resource allocation matrix based on the current computing cluster state; measuring bandwidth and probing latency of communication links between computing nodes; generating a topology constraint rule set based on communication performance indicators; determining the batch size adjustment range based on the computational characteristics of the training task; constructing an initial decision table including node identifiers, resource quotas, and link restrictions; mapping the priority score to resource allocation weights; updating the resource quota values ​​in the initial decision table according to the resource allocation weights; writing the upper and lower limits of the batch size and adjustment step size parameters; and generating the final scheduling decision table.

4. The adaptive priority scheduling method for large model training tasks according to claim 1, characterized in that, The step of sending the scheduling decision table to the resource manager for execution and monitoring the execution process to generate an execution log containing resource allocation status, task migration records, and training progress indicators includes: parsing the scheduling decision table into a set of scheduling instructions executable by the resource manager; determining the task deployment order according to node affinity rules; issuing resource allocation instructions according to a preset resource change step size; starting the execution status collector to obtain computing node resource allocation data, task migration operation records, and training iteration counter values; sampling and aggregating the computing node resource allocation data; statistically analyzing the migration success rate and time distribution in the task migration operation records; calculating the training speed indicator based on the training iteration counter value; and writing the sampling results, statistical data, and calculation indicators into the execution log.

5. The adaptive priority scheduling method for large model training tasks according to claim 1, characterized in that, The process of adjusting the prediction model parameters based on the execution log, updating the scheduling decision table under the constraints of checkpoint protection period and training stability, and recording task priority change information includes: extracting time-series data of resource usage and actual training speed records from the execution log, calculating the deviation vector between the prediction model output value and the measured value, updating the prediction model parameters using an online learning method, recalculating the predicted resource demand value based on the updated prediction model, comparing the predicted resource demand value with the timestamp of the checkpoint to determine whether the protection period constraint is met, calculating the training speed fluctuation amplitude to verify the training stability index, updating the resource configuration parameters in the scheduling decision table when the constraint is met, and generating a priority change record containing the adjustment time point, the reason for the change, and the scope of impact.

6. An adaptive priority scheduling device for large model training tasks, characterized in that, The device includes: a resource requirement determination module, used to collect training task operation index data, construct a performance feature matrix containing computation step time and parameter synchronization delay, identify training stages and convergence states based on the performance feature matrix, generate a task state vector containing remaining workload, expected completion time, and computational resource utilization, train a prediction model based on the task state vector, and output a resource requirement prediction curve within a future time window, including: obtaining data streams of computation step time, parameter synchronization delay, video memory usage, and device computational load from the training task operation environment; performing time-series grouping and aggregation processing on the data streams; calculating the mean, variance, and trend slope of the indicators; constructing a performance index matrix containing multi-dimensional features; extracting training steps, current batch size, and model parameter quantity to generate basic feature vectors; calculating training stage discrimination markers based on the performance index matrix; calculating the remaining training workload in combination with the basic feature vectors; estimating the expected completion time based on the training progress curve; accumulating and averaging the computational device utilization over a time window; combining the stage discrimination markers, remaining workload, expected completion time, and accumulated average utilization to generate a task state vector; constructing a training sample set containing historical task state vector sequences and corresponding resource usage records, and using a sliding window method. The training sample set is segmented and labeled within a time window. A resource demand prediction model is trained based on the gradient descent algorithm and applied to the current task state vector. The original predicted values ​​output by the resource demand prediction model are smoothed to eliminate short-term fluctuations. A memory demand curve, a computational load curve, and a network bandwidth curve are generated by sampling at preset time intervals and then merged into a resource demand prediction curve. A resource scheduling decision module calculates the training benefit increment based on the resource demand prediction curve, estimates the task migration cost by combining checkpoint size and network topology, generates a priority score based on task urgency, the training benefit increment, the task migration cost, and computation node affinity, and constructs a scheduling decision table including resource allocation schemes, communication topology constraints, and batch dynamic adjustment strategies. The priority score is written into the scheduling decision table. A priority scheduling execution module distributes the scheduling decision table to the resource manager for execution, monitors the execution process, generates an execution log including resource allocation status, task migration records, and training progress indicators, adjusts the prediction model parameters based on the execution log, updates the scheduling decision table while satisfying checkpoint protection period and training stability constraints, and records task priority change information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the adaptive priority scheduling method for large model training tasks as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the adaptive priority scheduling method for large model training tasks as described in any one of claims 1 to 5.

Images