A heterogeneous data scheduling method and system for model training

Abstract

The application provides a model training heterogeneous data scheduling method and system, which comprises the following steps: constructing an operator dependent directed acyclic graph, synchronously reading full training samples, and anchoring endogenous heterogeneous attribute parameters in the operator dependent directed acyclic graph according to the full training samples; aggregating the operator execution units with parameter values of the endogenous heterogeneous attribute parameters within a preset parameter threshold into an operator scheduling batch, and synchronously binding the operator scheduling batch into a minimum scheduling execution unit without execution dependency conflicts; collecting the computing power topology parameters of each distributed computing node, synchronously performing static routing mapping of the minimum scheduling execution unit and the computing power topology parameters, and distributing corresponding target computing nodes; in the process of model iterative training, collecting the operator execution progress and resource occupation state of each target computing node, synchronously combining the operator execution parameters, and updating the endogenous heterogeneous attribute parameters of each training sample correspondingly. The application can effectively control the training cost.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a heterogeneous data scheduling method and system for model training. Background Technology

[0002] As the scale of AI models, such as large language models and multimodal models, continues to expand, distributed parallel training has become the mainstream implementation method for model training. During the training process, a large amount of heterogeneous data with significant differences in structure, lifecycle, and bandwidth sensitivity, such as model parameters, gradient tensors, intermediate activation values, and optimizer states, are generated endogenously. Existing heterogeneous data scheduling technologies are mostly focused on goals such as optimizing memory usage, ensuring distributed data consistency, and controlling training convergence accuracy. Relatively mature basic technology systems have been formed, including sharding scheduling, mixed precision adaptation, and hot / cold data stratification.

[0003] However, existing heterogeneous data scheduling technologies all assume that the computing power and network performance of cluster nodes are homogeneous, and do not design slow node adaptation mechanisms for the differentiated characteristics of heterogeneous data. Faced with the slow node problem that is common in large-scale clusters due to hardware performance fluctuations, network bandwidth jitter, and uneven IO load, the industry generally adopts post-event remedial task scheduling solutions such as task migration, redundant computing, and static load balancing. Such solutions not only cause more than 30% of computing power to be wasted, but also easily lead to problems such as gradient expiration and parameter synchronization misalignment, affecting the model convergence accuracy, and failing to eliminate the cluster-wide waiting and blocking effect caused by slow nodes at the root.

[0004] Currently, there is no mature technical solution in the industry that addresses the differentiated characteristics of heterogeneous data inherent in model training and resolves the long-tail latency problem of slow nodes from the perspective of data scheduling. This has led to a significant increase in the time and computing power costs of training large models, which has become a core industry bottleneck restricting the improvement of the training efficiency of large-scale AI models. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a heterogeneous data scheduling method and system for model training, so as to solve the problem that there is no mature technical solution in the existing technology to address the differentiated characteristics of the endogenous heterogeneous data in model training and to resolve the long-tail latency problem of slow nodes from the data scheduling dimension, which leads to an increase in the time cost and computing power cost of model training.

[0006] The first aspect of the present invention proposes: A method for scheduling heterogeneous data for model training, wherein the method includes: The forward and backward propagation computation links of the model to be trained are decomposed into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph. The full training samples are read out simultaneously, and the endogenous heterogeneous attribute parameters corresponding to each operator execution unit are anchored in the operator-dependent directed acyclic graph according to the full training samples. According to the execution timing constraints of the operator-dependent directed acyclic graph, the operator execution units whose parameter values ​​of the endogenous heterogeneous attribute parameters are within the preset parameter threshold are aggregated into operator scheduling batches. Simultaneously, the operator scheduling batches with upstream and downstream execution dependencies are bound to the smallest scheduling execution unit without execution dependency conflicts. Collect the computing power topology parameters of each distributed computing node, and synchronously perform static routing mapping between the minimum scheduling execution unit and the computing power topology parameters to allocate the corresponding target computing node to the minimum scheduling execution unit; During the model iterative training process, the minimum scheduling execution unit is deployed inside the corresponding target computing node to collect the operator execution progress and resource occupancy status of each target computing node. Simultaneously, combined with the operator execution parameters, the endogenous heterogeneous attribute parameters of each training sample are updated to complete the scheduling of heterogeneous data.

[0007] The beneficial effects of this invention are as follows: This technical solution decomposes the model computation link into operator execution units and constructs an operator-dependent directed acyclic graph, achieving precise pre-anchoring of the intrinsic heterogeneous attributes of training samples, and matching the differentiated processing needs of heterogeneous data from the data scheduling dimension; by binding operator scheduling batch aggregation with the smallest scheduling execution unit, scheduling timing disorder caused by upstream and downstream operator dependency conflicts is avoided; through static routing mapping of computing power topology parameters, precise matching of computing power between operator units and distributed computing nodes is achieved, fundamentally resolving the long-tail latency pain point of slow nodes; at the same time, by dynamically updating heterogeneous attribute parameters with execution data during the training process, closed-loop iterative optimization of the scheduling strategy is achieved, significantly compressing the overall model training time, significantly reducing redundant computing power consumption, and effectively controlling the time and computing power costs of model training.

[0008] Furthermore, the step of decomposing the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct a corresponding operator-dependent directed acyclic graph, includes: The computational features of the forward and backward propagation links of the model to be trained are scanned to identify the corresponding heterogeneous scheduling adaptation boundary. Based on the heterogeneous scheduling adaptation boundary, the forward and backward propagation links are synchronously and collaboratively decomposed to generate several operator execution units. Using each of the operator execution units as graph nodes and data flow dependency identifiers as the basis, directed dependency edges are constructed accordingly. Simultaneously, an initial operator dependency directed acyclic graph is formed based on the directed dependency edges. Here, heterogeneous attribute labels are embedded for each of the graph nodes, and a dependency strength quantization value is assigned to each of the directed dependency edges. Based on the temporal constraints of forward inference, backward gradient calculation, and weight update during model training, the initial operator-dependent directed acyclic graph is topologically sorted and layered. After simultaneous full-link acyclicity and temporal compliance verification, the corresponding operator-dependent directed acyclic graph is generated.

[0009] Furthermore, the step of reading the full training samples and anchoring the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph based on the full training samples includes: Based on the full training samples read, the full statistics of sample dimension, sparsity, numerical distribution and feature fluctuation amplitude are completed to generate a sample heterogeneous feature profile. Simultaneously, based on the sample heterogeneous feature profile, the sample feature transmission link corresponding to the input and output ports of the full-link operator is mapped in the operator-dependent directed acyclic graph. Based on the sample feature transmission link, the sample feature processing rules and transmission paths of each operator execution unit are identified, so as to construct the corresponding endogenous association link between the sample and the operator. Based on the execution timing constraints of the operator-dependent directed acyclic graph, similarity clustering is performed on the parameter fluctuation range of operators in the same layer according to the endogenous association links, and the parameter matching degree of cross-layer upstream and downstream operators is verified. Operator parameters with scheduling adaptation conflict risks are marked and corrected simultaneously to anchor the endogenous heterogeneous attribute parameters accordingly.

[0010] Furthermore, the step of performing static route mapping between the minimum scheduling execution unit and the computing power topology parameters to allocate the corresponding target computing node to the minimum scheduling execution unit includes: Based on the hardware architecture, communication topology, and heterogeneous adaptation capabilities, the computing power topology parameters are divided into several heterogeneous computing power domains. Simultaneously, based on the execution timing of the operator-dependent directed acyclic graph, the smallest scheduling execution units that are continuously bound upstream and downstream are aggregated into corresponding scheduling execution groups. Extract the heterogeneous attribute aggregation features, temporal constraints, and resource requirement baselines of each scheduling execution group to construct the adaptation and matching rules between the scheduling execution group and the heterogeneous computing power domain. Simultaneously, based on the adaptation and matching rules, according to the temporal priority and attribute matching degree of the smallest scheduling execution unit, match the corresponding candidate computing nodes within the corresponding heterogeneous computing power domain. The candidate computing nodes are dynamically filtered to output the target computing node.

[0011] Furthermore, the step of dynamically filtering the candidate computing nodes to output the target computing node includes: Based on the endogenous heterogeneous attribute parameters, the corresponding heterogeneous attribute dynamic drift trajectory generated by the minimum scheduling execution unit in the complete model iteration training cycle is simulated, and the heterogeneous attribute dynamic drift trajectory is simultaneously decomposed into the resource demand fluctuation range and time-series constraint drift threshold of the whole cycle training. Based on the resource demand fluctuation range and the time-series constraint drift threshold, a full-iteration collaborative bearing simulation is performed on each of the candidate computing nodes to output the corresponding pre-matched node combination; The pre-matched nodes are combined to perform full-link compliance verification, and the target computing nodes are selected accordingly based on the verification results.

[0012] Furthermore, the step of deploying the minimum scheduling execution unit within its corresponding target computing node during model iterative training, in order to collect the operator execution progress and resource usage status of each target computing node, includes: Set corresponding resource consumption floating upper limit and execution timing fault tolerance window for each target computing node to construct a corresponding node execution monitoring baseline. Simultaneously, when deploying the minimum scheduling execution unit, allocate corresponding acquisition probes to each operator execution unit in the minimum scheduling execution unit. The acquisition probe is used to obtain the actual working parameters of the corresponding operator execution unit. Based on the actual working parameters, cross-node timing alignment and association marking processing is performed on the upstream and downstream bound target computing nodes to obtain the corresponding timing acquisition dataset. The time-series acquisition dataset is subjected to feature aggregation processing to generate corresponding execution feature vectors. Simultaneously, the execution feature vectors are converted in real time to generate the operator execution progress and resource occupancy status.

[0013] Furthermore, the step of performing real-time transformation processing on the execution feature vector to generate the corresponding operator execution progress and resource occupancy status includes: The execution feature vector is bound to each of the acquisition probes to extract the edge weight parameters of the operator-dependent directed acyclic graph. The edge weight parameters are used as check bits and embedded into the header of the bound execution feature vector to output the corresponding target feature vector. Extract the time-series fault-tolerant window parameters of the node execution monitoring baseline, and synchronously slice the target feature vector according to the time interval of the time-series fault-tolerant window parameters to obtain several time-series feature segments; Each of the time-series feature segments is matched with the type of the endogenous heterogeneous attribute parameter. The successfully matched time-series feature segments are used as the progress data of the operator execution progress, and the unmatched time-series feature segments are used as the status data of the resource occupancy status.

[0014] The second aspect of the present invention proposes: A heterogeneous data scheduling system for model training, wherein the system comprises: The construction module is used to decompose the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph, and synchronously read out the full training samples, so as to anchor the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph according to the full training samples. The processing module is used to aggregate operator execution units whose parameter values ​​of the endogenous heterogeneous attribute parameters are within a preset parameter threshold into operator scheduling batches according to the execution timing constraints of the operator-dependent directed acyclic graph, and simultaneously bind operator scheduling batches with upstream and downstream execution dependencies to the smallest scheduling execution unit without execution dependency conflicts. The mapping module is used to collect the computing power topology parameters of each distributed computing node and synchronously perform static route mapping between the minimum scheduling execution unit and the computing power topology parameters, so as to allocate the corresponding target computing node to the minimum scheduling execution unit. The scheduling module is used to deploy the smallest scheduling execution unit inside the corresponding target computing node during the model iterative training process, so as to collect the operator execution progress and resource occupancy status of each target computing node, and simultaneously update the endogenous heterogeneous attribute parameters of each training sample in combination with the operator execution parameters, so as to complete the scheduling of heterogeneous data.

[0015] Furthermore, the building module is specifically used for: The computational features of the forward and backward propagation links of the model to be trained are scanned to identify the corresponding heterogeneous scheduling adaptation boundary. Based on the heterogeneous scheduling adaptation boundary, the forward and backward propagation links are synchronously and collaboratively decomposed to generate several operator execution units. Using each of the operator execution units as graph nodes and data flow dependency identifiers as the basis, directed dependency edges are constructed accordingly. Simultaneously, an initial operator dependency directed acyclic graph is formed based on the directed dependency edges. Here, heterogeneous attribute labels are embedded for each of the graph nodes, and a dependency strength quantization value is assigned to each of the directed dependency edges. Based on the temporal constraints of forward inference, backward gradient calculation, and weight update during model training, the initial operator-dependent directed acyclic graph is topologically sorted and layered. After simultaneous full-link acyclicity and temporal compliance verification, the corresponding operator-dependent directed acyclic graph is generated.

[0016] Furthermore, the building module is specifically used for: Based on the full training samples read, the full statistics of sample dimension, sparsity, numerical distribution and feature fluctuation amplitude are completed to generate a sample heterogeneous feature profile. Simultaneously, based on the sample heterogeneous feature profile, the sample feature transmission link corresponding to the input and output ports of the full-link operator is mapped in the operator-dependent directed acyclic graph. Based on the sample feature transmission link, the sample feature processing rules and transmission paths of each operator execution unit are identified, so as to construct the corresponding endogenous association link between the sample and the operator. Based on the execution timing constraints of the operator-dependent directed acyclic graph, similarity clustering is performed on the parameter fluctuation range of operators in the same layer according to the endogenous association links, and the parameter matching degree of cross-layer upstream and downstream operators is verified. Operator parameters with scheduling adaptation conflict risks are marked and corrected simultaneously to anchor the endogenous heterogeneous attribute parameters accordingly.

[0017] Furthermore, the mapping module is specifically used for: Based on the hardware architecture, communication topology, and heterogeneous adaptation capabilities, the computing power topology parameters are divided into several heterogeneous computing power domains. Simultaneously, based on the execution timing of the operator-dependent directed acyclic graph, the smallest scheduling execution units that are continuously bound upstream and downstream are aggregated into corresponding scheduling execution groups. Extract the heterogeneous attribute aggregation features, temporal constraints, and resource requirement baselines of each scheduling execution group to construct the adaptation and matching rules between the scheduling execution group and the heterogeneous computing power domain. Simultaneously, based on the adaptation and matching rules, according to the temporal priority and attribute matching degree of the smallest scheduling execution unit, match the corresponding candidate computing nodes within the corresponding heterogeneous computing power domain. The candidate computing nodes are dynamically filtered to output the target computing node.

[0018] Furthermore, the mapping module is specifically used for: Based on the endogenous heterogeneous attribute parameters, the corresponding heterogeneous attribute dynamic drift trajectory generated by the minimum scheduling execution unit in the complete model iteration training cycle is simulated, and the heterogeneous attribute dynamic drift trajectory is simultaneously decomposed into the resource demand fluctuation range and time-series constraint drift threshold of the whole cycle training. Based on the resource demand fluctuation range and the time-series constraint drift threshold, a full-iteration collaborative bearing simulation is performed on each of the candidate computing nodes to output the corresponding pre-matched node combination; The pre-matched nodes are combined to perform full-link compliance verification, and the target computing nodes are selected accordingly based on the verification results.

[0019] Furthermore, the scheduling module is specifically used for: Set corresponding resource consumption floating upper limit and execution timing fault tolerance window for each target computing node to construct a corresponding node execution monitoring baseline. Simultaneously, when deploying the minimum scheduling execution unit, allocate corresponding acquisition probes to each operator execution unit in the minimum scheduling execution unit. The acquisition probe is used to obtain the actual working parameters of the corresponding operator execution unit. Based on the actual working parameters, cross-node timing alignment and association marking processing is performed on the upstream and downstream bound target computing nodes to obtain the corresponding timing acquisition dataset. The time-series acquisition dataset is subjected to feature aggregation processing to generate corresponding execution feature vectors. Simultaneously, the execution feature vectors are converted in real time to generate the operator execution progress and resource occupancy status.

[0020] Furthermore, the scheduling module is specifically used for: The execution feature vector is bound to each of the acquisition probes to extract the edge weight parameters of the operator-dependent directed acyclic graph. The edge weight parameters are used as check bits and embedded into the header of the bound execution feature vector to output the corresponding target feature vector. Extract the time-series fault-tolerant window parameters of the node execution monitoring baseline, and synchronously slice the target feature vector according to the time interval of the time-series fault-tolerant window parameters to obtain several time-series feature segments; Each of the time-series feature segments is matched with the type of the endogenous heterogeneous attribute parameter. The successfully matched time-series feature segments are used as the progress data of the operator execution progress, and the unmatched time-series feature segments are used as the status data of the resource occupancy status.

[0021] The third aspect of the present invention proposes: A computer includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a heterogeneous data scheduling method for model training as described above.

[0022] The fourth aspect of the present invention proposes: A readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements a heterogeneous data scheduling method for model training as described above.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] Figure 1 A flowchart of a heterogeneous data scheduling method for model training provided in the first embodiment of the present invention; Figure 2 The structural block diagram of the heterogeneous data scheduling system for model training provided in the third embodiment of the present invention is shown.

[0025] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0026] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0027] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] Please see Figure 1The diagram illustrates a heterogeneous data scheduling method for model training provided in the first embodiment of the present invention. This method enables precise matching of computing power between operator units and distributed computing nodes through static routing mapping of computing power topology parameters, thereby addressing the long-tail latency problem of slow nodes at its source. Simultaneously, it dynamically updates heterogeneous attribute parameters based on the execution data during the training process, achieving closed-loop iterative optimization of the scheduling strategy. This significantly reduces the overall time consumption of model training, substantially decreases redundant computing power consumption, and effectively controls the time and computing power costs of model training.

[0030] Specifically, this embodiment provides: A method for scheduling heterogeneous data for model training, wherein the method includes: Step S10: Decompose the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph. Simultaneously read out the full training samples, so as to anchor the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph according to the full training samples. It should be noted that traditional scheduling methods typically only decompose the forward propagation path of the model, completely severing the strong dependency between forward inference, backward gradient calculation, and weight updates, which can easily lead to execution timing conflicts. At the same time, they only use hardware architecture differences as the sole basis for heterogeneous scheduling, completely ignoring the core heterogeneous source of model training. Specifically, the intrinsic heterogeneous characteristics between the operators themselves and the training samples, that is, the different resource requirements brought by the different operator's processing rules for sample features, the different dimensions, sparsity, and numerical distribution of the samples to the operator computation, are the core root cause of the mismatch between scheduling and actual computational needs. This step first decomposes the complete computational chain of the model to be trained—forward propagation, backward gradient calculation, and weight update—into operator execution units with explicit execution dependencies. This ensures that the decomposed units fully cover the entire computational process of model training and retain the original dependencies between the forward and backward links. Then, based on these operator execution units, a directed acyclic graph of operator dependencies is constructed to fully present the execution order and dependencies of all operators. Most importantly, this step introduces feature statistics of the entire training sample. Based on the heterogeneous characteristics of the training samples, the intrinsic heterogeneous attribute parameters corresponding to each operator execution unit are anchored in the directed acyclic graph of operator dependencies. These parameters directly reflect the real computational resource requirements, timing constraints, and data transmission overhead of different operators when processing the corresponding sample features. This grasps the core essence of model training heterogeneity from the source of scheduling, rather than just focusing on the superficial hardware heterogeneity.

[0031] Step S20: According to the execution timing constraints of the operator-dependent directed acyclic graph, the operator execution units whose parameter values ​​of the endogenous heterogeneous attribute parameters are within the preset parameter threshold are aggregated into operator scheduling batches. Simultaneously, the operator scheduling batches with upstream and downstream execution dependencies are bound to the smallest scheduling execution unit without execution dependency conflicts. It's important to note that traditional scheduling methods typically use single operators or single layers as scheduling units. This can lead to either excessively fine granularity, resulting in exponentially increasing cross-node communication overhead, or excessively coarse granularity, causing dependency conflicts and insufficient parallelism. Furthermore, they fail to consider the similarity of inherent heterogeneous attributes between operators, scheduling operators with vastly different resource requirements to the same node, leading to unbalanced node resource utilization. This step, based on the execution timing constraints of a directed acyclic graph (DAG) of operator dependencies, first aggregates operator execution units with similar endogenous heterogeneous attribute parameters (parameter values ​​within a preset threshold) into operator scheduling batches. This ensures that the resource requirements and timing constraints of operators within the same batch are highly consistent, avoiding resource mismatch issues within the same node. Then, operator scheduling batches with strict upstream and downstream execution dependencies are bound into the smallest scheduling execution unit without execution dependency conflicts. This smallest unit has two core characteristics: first, the execution timing of operators within the unit is completely closed-loop, with no cross-dependencies with operators outside the unit; second, upstream and downstream operators within the unit can achieve pipelined parallel execution, minimizing cross-node communication overhead and waiting latency. This aggregation and binding not only resolves the execution dependency conflicts between operators, but also achieves an optimal balance in scheduling granularity, taking into account both training parallelism and communication efficiency.

[0032] Step S30: Collect the computing power topology parameters of each distributed computing node, and synchronously perform static route mapping between the minimum scheduling execution unit and the computing power topology parameters to allocate the corresponding target computing node to the minimum scheduling execution unit; It's important to note that traditional scheduling methods typically treat distributed computing nodes as a homogeneous resource pool, randomly scheduling based solely on currently available resources. This completely ignores the differences in hardware architecture, communication topology, and heterogeneous acceleration capabilities among different computing nodes. This leads to the core problem of "computing power mismatch," where operators requiring high parallel computing power are scheduled to CPU nodes, and operators requiring large amounts of memory are scheduled to memory-constrained nodes. This step first collects complete computing power topology parameters for all distributed computing nodes, including hardware architecture (CPU / GPU / NPU / heterogeneous accelerator card), number of cores, memory / GPU capacity, communication bandwidth, inter-node network topology, and heterogeneous operator adaptability—a comprehensive set of parameters to fully reconstruct the computing power topology of the distributed cluster. Then, it performs static routing mapping between the inherent heterogeneous attributes, resource requirements, and timing constraints of the smallest scheduling execution unit and the computing power topology parameters. This means that for each smallest scheduling execution unit, a computing node with a fully compatible hardware architecture, computing power, and communication conditions is matched. This precise matching of operators and computing power is achieved before training begins, avoiding frequent scheduling migrations and resource contention during training.

[0033] Step S40: During the model iterative training process, the minimum scheduling execution unit is deployed inside the corresponding target computing node to collect the operator execution progress and resource occupancy status of each target computing node. Simultaneously, combined with the operator execution parameters, the endogenous heterogeneous attribute parameters of each training sample are updated to complete the scheduling of heterogeneous data.

[0034] It's important to note that traditional static scheduling methods fix the mapping between operators and nodes after training begins, making them completely incompatible with the drift of intrinsic heterogeneous attributes during model training. Specifically, as the model iterates, the weight distribution of operators, the feature processing logic of samples, and the complexity of gradient calculations all dynamically change, causing the initial scheduling match to gradually become ineffective, leading to a sharp decline in efficiency in the later stages of training. This step, throughout the entire model training cycle, continuously collects real-time runtime data such as operator execution progress and resource usage status of the target computing nodes. It compares the actual execution status with the initial intrinsic heterogeneous attribute parameters, dynamically updates the intrinsic heterogeneous attribute parameters corresponding to the training samples and operators based on the actual execution parameters, and then dynamically optimizes and adjusts the scheduling strategy based on the updated parameters. This achieves a complete scheduling closed loop from static pre-matching to dynamic closed-loop optimization, ensuring that the match between operators and computing power remains optimal throughout the entire model training cycle, completely solving the adaptability failure problem of traditional static scheduling.

[0035] Second Embodiment Furthermore, the step of decomposing the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct a corresponding operator-dependent directed acyclic graph, includes: The computational features of the forward and backward propagation links of the model to be trained are scanned to identify the corresponding heterogeneous scheduling adaptation boundary. Based on the heterogeneous scheduling adaptation boundary, the forward and backward propagation links are synchronously and collaboratively decomposed to generate several operator execution units. Using each of the operator execution units as graph nodes and data flow dependency identifiers as the basis, directed dependency edges are constructed accordingly. Simultaneously, an initial operator dependency directed acyclic graph is formed based on the directed dependency edges. Here, heterogeneous attribute labels are embedded for each of the graph nodes, and a dependency strength quantization value is assigned to each of the directed dependency edges. Based on the temporal constraints of forward inference, backward gradient calculation, and weight update during model training, the initial operator-dependent directed acyclic graph is topologically sorted and layered. After simultaneous full-link acyclicity and temporal compliance verification, the corresponding operator-dependent directed acyclic graph is generated.

[0036] It should be noted that traditional decomposition methods only decompose the forward inference layer of the model, while the gradient calculation link of the backpropagation is automatically generated by the training framework without any decomposition and adaptation at the scheduling level. This results in a complete disconnect between the execution timing of operators in the forward and backward links, which can easily lead to problems such as waiting for gradient calculation and blocking of weight updates. At the same time, it is also impossible to reasonably schedule the operators of the backward link. This step first involves a complete scan of the entire computational chain of the model to be trained, including forward propagation, backward gradient calculation, and weight update. This scan identifies the adaptation boundaries for heterogeneous scheduling, determining which operators can be merged for scheduling, which operators must be executed independently, and which operators have strong cross-link dependencies. This clarifies the granularity and scope of the decomposition. Based on these adaptation boundaries, the forward and backward propagation chains are then synchronously and collaboratively decomposed. This ensures that each operator in the forward propagation chain forms a one-to-one correspondence with its corresponding backward gradient calculation operator. Each decomposed operator execution unit retains its execution dependencies throughout the entire chain, fundamentally preventing the separation of the forward and backward chains and ensuring that subsequent scheduling can cover the entire computational process of model training.

[0037] After decomposing the operator execution units, the scattered units need to be integrated into a complete execution logic graph through dependencies. This step treats each operator execution unit as an independent graph node and constructs directed dependency edges based on the data flow dependencies between operators. The direction of the directed edges strictly follows the data flow direction, i.e., from upstream operator to downstream operator, ensuring the correctness of the execution logic. Simultaneously, a unique heterogeneous attribute label is embedded in each graph node, marking basic heterogeneous information such as the operator's computation type, resource requirements, and compatible hardware type, providing a foundation for subsequent intrinsic heterogeneous attribute anchoring. A dependency strength quantification value is assigned to each directed dependency edge. This value represents the tightness of the dependency between upstream and downstream operators. For example, the dependency strength between the weight update operator and the gradient calculation operator is extremely high, while the dependency strength between two adjacent independent convolution operators is relatively low. The dependency strength quantification value provides a core basis for subsequent scheduling unit aggregation and communication overhead optimization.

[0038] The initially constructed directed acyclic graph (DAG) only presents the dependencies between operators, without specifying the execution order or hierarchical parallelism rules, and cannot be directly used for scheduling. This step strictly follows the inherent execution order of model training: first, forward inference is completed, then backward gradient calculation is performed, and finally, weight updates are performed. The initial operator-dependent DAG is topologically sorted to clarify the execution order of all operators. Then, according to the topological sorting result, hierarchical processing is performed, grouping operators without execution dependencies and capable of parallel execution into the same layer to maximize the parallel potential of model training. Subsequently, the sorted and hierarchical graph undergoes full-link acyclicity verification and timing compliance verification to completely eliminate problems such as circular dependencies and timing inversions in the graph, ensuring that the final generated operator-dependent DAG fully conforms to the execution logic of model training and possesses optimal parallel execution potential, providing an accurate and reliable execution blueprint for subsequent heterogeneous scheduling.

[0039] Furthermore, the step of reading the full training samples and anchoring the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph based on the full training samples includes: Based on the full training samples read, the full statistics of sample dimension, sparsity, numerical distribution and feature fluctuation amplitude are completed to generate a sample heterogeneous feature profile. Simultaneously, based on the sample heterogeneous feature profile, the sample feature transmission link corresponding to the input and output ports of the full-link operator is mapped in the operator-dependent directed acyclic graph. Based on the sample feature transmission link, the sample feature processing rules and transmission paths of each operator execution unit are identified, so as to construct the corresponding endogenous association link between the sample and the operator. Based on the execution timing constraints of the operator-dependent directed acyclic graph, similarity clustering is performed on the parameter fluctuation range of operators in the same layer according to the endogenous association links, and the parameter matching degree of cross-layer upstream and downstream operators is verified. Operator parameters with scheduling adaptation conflict risks are marked and corrected simultaneously to anchor the endogenous heterogeneous attribute parameters accordingly.

[0040] It's important to note that during model training, the actual computational resource requirements of operators are essentially determined by the features of the samples they process: high-dimensional, highly sparsity samples impose significantly higher computational and memory overhead on the embedding layer and matrix multiplication operators compared to regular samples; samples with large feature fluctuations lead to a sharp increase in the complexity of gradient calculation operators. These are all endogenous sources of heterogeneity that traditional scheduling completely ignores. This step first performs a full-dimensional statistical analysis of all training samples, covering all core features that affect operator computational complexity, such as sample dimensionality, sparsity, numerical distribution, feature fluctuation amplitude, and outlier ratio, generating a complete profile of sample heterogeneity features and accurately reconstructing the heterogeneous characteristics of the training samples. Then, based on this profile of sample heterogeneity features, the entire link transmission path of sample features from the input layer to the output layer and then to the backward gradient calculation is mapped in the operator-dependent directed acyclic graph. This clarifies the sample feature characteristics corresponding to the input and output ports of each operator, that is, what features each operator is processing and what the heterogeneous characteristics of these features are, providing a precise path for the subsequent construction of endogenous correlation links.

[0041] After clarifying the end-to-end transmission path of sample features, this step further defines the processing rules for sample features by each operator execution unit, such as the feature extraction rules for convolution operators, the downsampling rules for pooling operators, the matrix mapping rules for fully connected layers, and the backpropagation rules for gradient operators. This clarifies the transformation logic of each operator on the input sample features and the impact of this transformation on the operator's computational complexity. Based on the processing rules and transmission paths, an endogenous correlation link is constructed that binds samples and operators one-to-one. This link fully presents the complete transmission logic of "sample heterogeneous characteristics → operator processing rules → operator computational resource requirements," allowing the resource requirements of each operator to be traced back to the corresponding sample feature characteristics. This completely breaks down the connection between sample heterogeneity and operator scheduling, making scheduling no longer a "blind scheduling" detached from training data, but a precise scheduling based entirely on the real needs of the training samples.

[0042] After constructing the endogenous correlation link, it is necessary to transform the dispersed heterogeneous characteristics of operators into standardized endogenous heterogeneous attribute parameters that can be used for scheduling. This step, based on the execution timing constraints of the operator-dependent directed acyclic graph, first clusters parallel-executable operators at the same layer based on the parameter fluctuation ranges corresponding to their intrinsic links. This ensures that the clustered operators have similar resource requirements and timing constraints, providing a basis for subsequent operator aggregation. Next, the parameter matching degree of cross-layer upstream and downstream operators is verified, focusing on the differences in resource requirements, data transmission volume, and dependency strength between upstream and downstream operators. Operators that may cause scheduling adaptation conflicts are identified, such as situations where the upstream operator outputs a very large amount of data and the downstream operator's memory resource requirements are mismatched. The corresponding operator parameters are then corrected. Finally, standardized intrinsic heterogeneous attribute parameters corresponding to each operator execution unit are anchored. These parameters include comprehensive information such as the operator's computational complexity, peak memory / GPU memory requirements, communication bandwidth requirements, timing constraint thresholds, sample feature sensitivity, and hardware adaptation preferences. This fully reflects the operator's actual scheduling requirements when processing corresponding sample features and serves as the core benchmark for the entire heterogeneous scheduling method.

[0043] Furthermore, the step of performing static route mapping between the minimum scheduling execution unit and the computing power topology parameters to allocate the corresponding target computing node to the minimum scheduling execution unit includes: Based on the hardware architecture, communication topology, and heterogeneous adaptation capabilities, the computing power topology parameters are divided into several heterogeneous computing power domains. Simultaneously, based on the execution timing of the operator-dependent directed acyclic graph, the smallest scheduling execution units that are continuously bound upstream and downstream are aggregated into corresponding scheduling execution groups. Extract the heterogeneous attribute aggregation features, temporal constraints, and resource requirement baselines of each scheduling execution group to construct the adaptation and matching rules between the scheduling execution group and the heterogeneous computing power domain. Simultaneously, based on the adaptation and matching rules, according to the temporal priority and attribute matching degree of the smallest scheduling execution unit, match the corresponding candidate computing nodes within the corresponding heterogeneous computing power domain. The candidate computing nodes are dynamically filtered to output the target computing node.

[0044] It should be noted that traditional scheduling methods treat all distributed computing nodes as a unified resource pool, completely ignoring the differences in hardware architecture of nodes in the cluster, the distance of network communication topology, and the heterogeneous adaptability to different types of operators. This results in extremely high cross-domain communication overhead and extremely poor operator-hardware compatibility. This step first uses the collected full-cluster computing power topology parameters to divide the entire cluster's computing nodes into several heterogeneous computing power domains that are internally homogeneous and externally differentiated, based on hardware architecture (such as GPU computing power domain, CPU computing power domain, and NPU heterogeneous acceleration domain), inter-node communication topology (such as the same rack domain under the same switch and the wide area domain across data centers), and heterogeneous adaptation capabilities for different types of operators. This ensures that nodes within the same computing power domain have similar hardware capabilities, communication latency, and adaptation characteristics, defining the scope for subsequent accurate matching. Simultaneously, based on the execution sequence of the operator-dependent directed acyclic graph, the smallest scheduling execution units with continuous upstream and downstream binding relationships and pipelined execution are aggregated into corresponding scheduling execution groups. Units within the same scheduling execution group have strong temporal dependencies and require high-frequency cross-node communication. Aggregating them allows for unified computing power matching, minimizing cross-computing power domain communication overhead.

[0045] After completing the partitioning of computing power domains and aggregation of scheduling execution groups, it is necessary to construct scientific adaptation and matching rules, rather than simply matching resources in idle capacity. This step first extracts the heterogeneous attribute aggregation features of each scheduling execution group, which is the aggregation result of the endogenous heterogeneous attribute parameters of all the smallest scheduling execution units within the group. Simultaneously, it clarifies the strict timing constraints and resource requirement baselines, including core indicators such as peak computing power requirements, memory / GPU memory requirements, communication bandwidth requirements, and maximum tolerable execution latency. Based on these indicators, it constructs adaptation and matching rules between scheduling execution groups and heterogeneous computing power domains. The core of these rules is "hard constraint priority, soft constraint optimization": first, computing power domains that cannot meet the timing constraints and resource requirement baselines are filtered out. Then, among the remaining compliant computing power domains, the computing power domain with the highest heterogeneous attribute aggregation feature adaptation degree is prioritized for matching. For example, scheduling execution groups mainly engaged in matrix calculations are prioritized for matching the GPU computing power domain, and scheduling execution groups mainly engaged in logic processing are prioritized for matching the CPU computing power domain. Subsequently, based on the adaptation and matching rules, according to the timing priority of the smallest scheduling execution unit (units on the critical path have higher priority) and attribute matching degree, candidate computing nodes that meet the requirements are matched within the corresponding heterogeneous computing power domains, providing a candidate pool for the subsequent final selection.

[0046] After obtaining the candidate computing node pool, it is necessary to further screen the optimal target computing node. This step involves dynamically screening the candidate computing nodes by comprehensively considering multiple dimensions such as their current resource idle status, communication latency between nodes, efficiency of executing similar operators in the past, and resource stability throughout the training cycle. Nodes with large resource fluctuations, high communication latency, and low historical execution efficiency are eliminated. Finally, the target computing node with the highest adaptability and strongest stability to the minimum scheduling execution unit is selected, completing the final step of static route mapping. This ensures that each minimum scheduling execution unit can be deployed on the computing node most suitable for its execution, achieving optimal matching of operators and computing power from before training starts.

[0047] Furthermore, the step of dynamically filtering the candidate computing nodes to output the target computing node includes: Based on the endogenous heterogeneous attribute parameters, the corresponding heterogeneous attribute dynamic drift trajectory generated by the minimum scheduling execution unit in the complete model iteration training cycle is simulated, and the heterogeneous attribute dynamic drift trajectory is simultaneously decomposed into the resource demand fluctuation range and time-series constraint drift threshold of the whole cycle training. Based on the resource demand fluctuation range and the time-series constraint drift threshold, a full-iteration collaborative bearing simulation is performed on each of the candidate computing nodes to output the corresponding pre-matched node combination; The pre-matched nodes are combined to perform full-link compliance verification, and the target computing nodes are selected accordingly based on the verification results.

[0048] It's important to note that as the model iterates through training, the operator weight distribution is constantly updated, and the processing logic for sample features changes. Consequently, the corresponding endogenous heterogeneous attribute parameters will dynamically drift. For example, in the early stages of training, gradient fluctuations are large, resulting in high peak resource requirements for gradient calculation operators. In the later stages, as gradients stabilize, resource requirements decrease significantly. Traditional static scheduling completely ignores this drift, causing initially matched nodes to fail to adapt to the changed resource requirements in the later stages of training. This step, based on anchored endogenous heterogeneous attribute parameters and combined with the iterative patterns of model training and the changing trends of sample features, first simulates the dynamic drift trajectory of the endogenous heterogeneous attributes of the smallest scheduling execution unit within a complete training iteration cycle, clarifying the changes in operator resource requirements and temporal constraints at each iteration stage. Then, this drift trajectory is decomposed into the resource requirement fluctuation range for the entire training cycle—that is, the maximum and minimum values ​​of computing power, memory, and bandwidth required by the operator throughout the training process—as well as the temporal constraint drift threshold, which is the maximum tolerable upper limit of operator execution latency. This provides a full-cycle demand benchmark for subsequent node load simulations.

[0049] After obtaining the demand baseline for the entire training cycle, this step performs a full-iteration-cycle collaborative carrying simulation for each candidate computing node: simulating whether the candidate node can continuously meet the requirements of resource demand fluctuation range and the timing constraint drift threshold during the process of carrying the corresponding minimum scheduling execution unit throughout the entire training cycle. At the same time, it simulates the collaborative execution effect of multiple dependent scheduling execution units on the corresponding candidate nodes to verify whether cross-node communication latency will lead to execution timing conflicts. Through the full-cycle simulation, candidate nodes that are currently resource-sufficient but will experience resource bottlenecks and timing violations in the later stages of training are eliminated. Finally, the pre-matched node combination that can stably carry the corresponding scheduling execution unit throughout the entire training cycle is output.

[0050] After obtaining the pre-matched node combination, this step performs a full-link compliance verification. The verification includes multiple dimensions such as node resource isolation compliance, cluster scheduling rule compliance, cross-node communication network policy compliance, and data security compliance. This ensures that the selected nodes not only meet performance requirements but also comply with cluster management rules and security requirements. Only nodes that pass the full-link compliance verification are ultimately determined as target computing nodes, completing the final selection of static routing mapping. This full-cycle simulation + compliance verification selection method completely solves the "short-sighted" problem of traditional static scheduling, ensuring that the target computing node can provide stable and adaptable computing power support for the smallest scheduling execution unit throughout the entire model training cycle, avoiding node migration and training interruptions during the training process.

[0051] Furthermore, the step of deploying the minimum scheduling execution unit within its corresponding target computing node during model iterative training, in order to collect the operator execution progress and resource usage status of each target computing node, includes: Set corresponding resource consumption floating upper limit and execution timing fault tolerance window for each target computing node to construct a corresponding node execution monitoring baseline. Simultaneously, when deploying the minimum scheduling execution unit, allocate corresponding acquisition probes to each operator execution unit in the minimum scheduling execution unit. The acquisition probe is used to obtain the actual working parameters of the corresponding operator execution unit. Based on the actual working parameters, cross-node timing alignment and association marking processing is performed on the upstream and downstream bound target computing nodes to obtain the corresponding timing acquisition dataset. The time-series acquisition dataset is subjected to feature aggregation processing to generate corresponding execution feature vectors. Simultaneously, the execution feature vectors are converted in real time to generate the operator execution progress and resource occupancy status.

[0052] It's important to note that achieving accurate status data collection requires first defining the monitoring baseline and boundaries, and ensuring a one-to-one correspondence between the data collection granularity and the operator execution unit. This step begins by setting corresponding resource usage fluctuation limits and execution timing tolerance windows for each target computing node based on the resource requirements and timing constraints of its smallest scheduled execution unit. This establishes a node execution monitoring baseline, which is the core benchmark for determining whether the node's execution status is normal and whether resource bottlenecks or timing violations have occurred. Simultaneously, when deploying the smallest scheduled execution unit to the target computing node, a dedicated data collection probe is assigned to each operator execution unit within the unit. The probe's lifecycle is fully bound to the operator execution unit, enabling precise capture of the corresponding operator's entire lifecycle execution data. This completely solves the problem of traditional monitoring having coarse granularity and being unable to correspond to specific operators.

[0053] After the acquisition probe is deployed, it will continuously collect the actual working parameters of the corresponding operator execution unit throughout the entire lifecycle of the operator execution, including the operator's start time, execution duration, completion progress, peak computing resource usage, memory / GPU memory usage, data transfer volume, and other full-dimensional execution data. At the same time, based on the upstream and downstream dependencies in the operator dependency directed acyclic graph, the acquisition data of different target computing nodes bound to upstream and downstream nodes will be aligned and associated across nodes to ensure that the execution data of upstream and downstream operators are on the same global time-series benchmark and can accurately present the execution status of the dependency link. For example, the interval between the completion time of the upstream operator and the start time of the downstream operator, that is, the waiting latency across nodes, completely solves the problem of time-series breakage of dependency links caused by traditional single-node acquisition. Finally, all the acquired data after alignment and marking will be integrated into a time-series acquisition dataset that is continuous and associated.

[0054] After obtaining the time-series acquisition dataset, this step performs feature aggregation processing to extract core features that reflect the execution status and resource consumption of operators, generating a standardized execution feature vector. This vector fully contains core information such as the execution progress, timing compliance, resource consumption, and execution status of dependent links of the corresponding operator. Then, the execution feature vector is transformed in real time, converting the aggregated feature data into operator execution progress and resource consumption status that the scheduling system can directly identify and use for optimization decisions. This includes clear status information such as the completion percentage of each operator, whether it exceeds the timing fault tolerance window, whether the resource consumption exceeds the floating limit, and whether a resource bottleneck has occurred. This provides accurate and real-time input for subsequent updates of endogenous heterogeneous attribute parameters and optimization of scheduling strategies.

[0055] Furthermore, the step of performing real-time transformation processing on the execution feature vector to generate the corresponding operator execution progress and resource occupancy status includes: The execution feature vector is bound to each of the acquisition probes to extract the edge weight parameters of the operator-dependent directed acyclic graph. The edge weight parameters are used as check bits and embedded into the header of the bound execution feature vector to output the corresponding target feature vector. Extract the time-series fault-tolerant window parameters of the node execution monitoring baseline, and synchronously slice the target feature vector according to the time interval of the time-series fault-tolerant window parameters to obtain several time-series feature segments; Each of the time-series feature segments is matched with the type of the endogenous heterogeneous attribute parameter. The successfully matched time-series feature segments are used as the progress data of the operator execution progress, and the unmatched time-series feature segments are used as the status data of the resource occupancy status.

[0056] It's important to note that the core value of the execution feature vector lies in its ability to accurately map to specific nodes and dependencies in the operator-dependent directed acyclic graph (DAG). Otherwise, the collected data would be meaningless isolated values. This step first uniquely binds the execution feature vector to its corresponding acquisition probe. Then, using the association information from the acquisition probe, it extracts the edge weight parameters of the upstream and downstream directed edges corresponding to the operator in the operator-dependent DAG, which are the dependency strength quantification values. This edge weight parameter is then used as a checksum and embedded into the header of the bound execution feature vector to generate the target feature vector. This checksum serves two core functions: first, it verifies the matching between the execution feature vector and the corresponding operator and dependency, avoiding data misalignment; second, it provides a core basis for subsequent time-series slicing and dependency analysis, ensuring that the transformed data fully reflects the execution state of the operator in the dependency chain.

[0057] The execution progress and resource usage status of operators change dynamically over time. Continuous feature vectors cannot accurately reflect the execution status at different time stages, nor can they be matched and verified against the time-series fault-tolerance window of the monitoring baseline. This step first extracts the time-series fault-tolerance window parameters from the node execution monitoring baseline. Using the time interval of this window as a unit, the continuous target feature vector is sliced ​​at equal intervals to obtain several time-continuous feature segments of consistent length. Each segment corresponds to operator execution and resource usage data within a fixed time window. Through this slicing process, continuous execution data can be decomposed into discrete verification units that match the monitoring baseline, facilitating subsequent type matching and status verification, while also accurately capturing execution anomalies and resource fluctuations within a specific time window.

[0058] Endogenous heterogeneous attribute parameters are divided into two main types: one type is progress-related parameters directly related to operator execution timing and completion, such as operator execution time, computation completion rate, and timing constraint compliance; the other type is resource-related parameters directly related to node resource consumption, such as computing power utilization, memory / GPU memory consumption, and bandwidth consumption. This step, based on the classification of endogenous heterogeneous attribute parameters, performs matching processing on each time-series feature segment: features in the segment that successfully match with progress-related endogenous heterogeneous attribute parameters are classified as progress data of operator execution progress, directly reflecting the operator's execution completion status and timing compliance; feature segments that fail to match but correspond to resource-related parameters are classified as status data of resource consumption status, directly reflecting the node's resource usage and whether resource bottlenecks have occurred. Through this type matching, accurate separation and standardized output of execution progress data and resource consumption status data are achieved, ensuring that the converted data can be directly used for updating endogenous heterogeneous attribute parameters and dynamically optimizing scheduling strategies, completing the dynamic closed loop of the entire heterogeneous scheduling method.

[0059] Please see Figure 2 The third embodiment of the present invention provides: A heterogeneous data scheduling system for model training, wherein the system comprises: The construction module is used to decompose the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph, and synchronously read out the full training samples, so as to anchor the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph according to the full training samples. The processing module is used to aggregate operator execution units whose parameter values ​​of the endogenous heterogeneous attribute parameters are within a preset parameter threshold into operator scheduling batches according to the execution timing constraints of the operator-dependent directed acyclic graph, and simultaneously bind operator scheduling batches with upstream and downstream execution dependencies to the smallest scheduling execution unit without execution dependency conflicts. The mapping module is used to collect the computing power topology parameters of each distributed computing node and synchronously perform static route mapping between the minimum scheduling execution unit and the computing power topology parameters, so as to allocate the corresponding target computing node to the minimum scheduling execution unit. The scheduling module is used to deploy the smallest scheduling execution unit inside the corresponding target computing node during the model iterative training process, so as to collect the operator execution progress and resource occupancy status of each target computing node, and simultaneously update the endogenous heterogeneous attribute parameters of each training sample in combination with the operator execution parameters, so as to complete the scheduling of heterogeneous data.

[0060] Furthermore, the building module is specifically used for: The computational features of the forward and backward propagation links of the model to be trained are scanned to identify the corresponding heterogeneous scheduling adaptation boundary. Based on the heterogeneous scheduling adaptation boundary, the forward and backward propagation links are synchronously and collaboratively decomposed to generate several operator execution units. Using each of the operator execution units as graph nodes and data flow dependency identifiers as the basis, directed dependency edges are constructed accordingly. Simultaneously, an initial operator dependency directed acyclic graph is formed based on the directed dependency edges. Here, heterogeneous attribute labels are embedded for each of the graph nodes, and a dependency strength quantization value is assigned to each of the directed dependency edges. Based on the temporal constraints of forward inference, backward gradient calculation, and weight update during model training, the initial operator-dependent directed acyclic graph is topologically sorted and layered. After simultaneous full-link acyclicity and temporal compliance verification, the corresponding operator-dependent directed acyclic graph is generated.

[0061] Furthermore, the building module is specifically used for: Based on the full training samples read, the full statistics of sample dimension, sparsity, numerical distribution and feature fluctuation amplitude are completed to generate a sample heterogeneous feature profile. Simultaneously, based on the sample heterogeneous feature profile, the sample feature transmission link corresponding to the input and output ports of the full-link operator is mapped in the operator-dependent directed acyclic graph. Based on the sample feature transmission link, the sample feature processing rules and transmission paths of each operator execution unit are identified, so as to construct the corresponding endogenous association link between the sample and the operator. Based on the execution timing constraints of the operator-dependent directed acyclic graph, similarity clustering is performed on the parameter fluctuation range of operators in the same layer according to the endogenous association links, and the parameter matching degree of cross-layer upstream and downstream operators is verified. Operator parameters with scheduling adaptation conflict risks are marked and corrected simultaneously to anchor the endogenous heterogeneous attribute parameters accordingly.

[0062] Furthermore, the mapping module is specifically used for: Based on the hardware architecture, communication topology, and heterogeneous adaptation capabilities, the computing power topology parameters are divided into several heterogeneous computing power domains. Simultaneously, based on the execution timing of the operator-dependent directed acyclic graph, the smallest scheduling execution units that are continuously bound upstream and downstream are aggregated into corresponding scheduling execution groups. Extract the heterogeneous attribute aggregation features, temporal constraints, and resource requirement baselines of each scheduling execution group to construct the adaptation and matching rules between the scheduling execution group and the heterogeneous computing power domain. Simultaneously, based on the adaptation and matching rules, according to the temporal priority and attribute matching degree of the smallest scheduling execution unit, match the corresponding candidate computing nodes within the corresponding heterogeneous computing power domain. The candidate computing nodes are dynamically filtered to output the target computing node.

[0063] Furthermore, the mapping module is specifically used for: Based on the endogenous heterogeneous attribute parameters, the corresponding heterogeneous attribute dynamic drift trajectory generated by the minimum scheduling execution unit in the complete model iteration training cycle is simulated, and the heterogeneous attribute dynamic drift trajectory is simultaneously decomposed into the resource demand fluctuation range and time-series constraint drift threshold of the whole cycle training. Based on the resource demand fluctuation range and the time-series constraint drift threshold, a full-iteration collaborative bearing simulation is performed on each of the candidate computing nodes to output the corresponding pre-matched node combination; The pre-matched nodes are combined to perform full-link compliance verification, and the target computing nodes are selected accordingly based on the verification results.

[0064] Furthermore, the scheduling module is specifically used for: Set corresponding resource consumption floating upper limit and execution timing fault tolerance window for each target computing node to construct a corresponding node execution monitoring baseline. Simultaneously, when deploying the minimum scheduling execution unit, allocate corresponding acquisition probes to each operator execution unit in the minimum scheduling execution unit. The acquisition probe is used to obtain the actual working parameters of the corresponding operator execution unit. Based on the actual working parameters, cross-node timing alignment and association marking processing is performed on the upstream and downstream bound target computing nodes to obtain the corresponding timing acquisition dataset. The time-series acquisition dataset is subjected to feature aggregation processing to generate corresponding execution feature vectors. Simultaneously, the execution feature vectors are converted in real time to generate the operator execution progress and resource occupancy status.

[0065] Furthermore, the scheduling module is specifically used for: The execution feature vector is bound to each of the acquisition probes to extract the edge weight parameters of the operator-dependent directed acyclic graph. The edge weight parameters are used as check bits and embedded into the header of the bound execution feature vector to output the corresponding target feature vector. Extract the time-series fault-tolerant window parameters of the node execution monitoring baseline, and synchronously slice the target feature vector according to the time interval of the time-series fault-tolerant window parameters to obtain several time-series feature segments; Each of the time-series feature segments is matched with the type of the endogenous heterogeneous attribute parameter. The successfully matched time-series feature segments are used as the progress data of the operator execution progress, and the unmatched time-series feature segments are used as the status data of the resource occupancy status.

[0066] The fourth embodiment of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heterogeneous data scheduling method for model training as described above.

[0067] The fifth embodiment of the present invention provides a readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements the heterogeneous data scheduling method for model training as described above.

[0068] In summary, the heterogeneous data scheduling method and system for model training provided in the above embodiments of the present invention can achieve precise matching of computing power between operator units and distributed computing nodes through static routing mapping of computing power topology parameters, thereby resolving the long-tail latency problem of slow nodes from the root. At the same time, by dynamically updating heterogeneous attribute parameters with execution data during the training process, the scheduling strategy can be optimized in a closed loop, significantly reducing the overall model training time, significantly reducing redundant computing power consumption, and effectively controlling the time and computing power costs of model training.

[0069] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0070] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0071] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0072] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0073] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0074] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for scheduling heterogeneous data for model training, characterized in that, The method includes: The forward and backward propagation computation links of the model to be trained are decomposed into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph. The full training samples are read out simultaneously, and the endogenous heterogeneous attribute parameters corresponding to each operator execution unit are anchored in the operator-dependent directed acyclic graph according to the full training samples. According to the execution timing constraints of the operator-dependent directed acyclic graph, the operator execution units whose parameter values ​​of the endogenous heterogeneous attribute parameters are within the preset parameter threshold are aggregated into operator scheduling batches. Simultaneously, the operator scheduling batches with upstream and downstream execution dependencies are bound to the smallest scheduling execution unit without execution dependency conflicts. Collect the computing power topology parameters of each distributed computing node, and synchronously perform static routing mapping between the minimum scheduling execution unit and the computing power topology parameters to allocate the corresponding target computing node to the minimum scheduling execution unit; During the model iterative training process, the minimum scheduling execution unit is deployed inside the corresponding target computing node to collect the operator execution progress and resource occupancy status of each target computing node. Simultaneously, combined with the operator execution parameters, the endogenous heterogeneous attribute parameters of each training sample are updated to complete the scheduling of heterogeneous data.

2. The heterogeneous data scheduling method for model training according to claim 1, characterized in that, The step of decomposing the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph, includes: The computational features of the forward and backward propagation links of the model to be trained are scanned to identify the corresponding heterogeneous scheduling adaptation boundary. Based on the heterogeneous scheduling adaptation boundary, the forward and backward propagation links are synchronously and collaboratively decomposed to generate several operator execution units. Using each of the operator execution units as graph nodes and data flow dependency identifiers as the basis, directed dependency edges are constructed accordingly. Simultaneously, an initial operator dependency directed acyclic graph is formed based on the directed dependency edges. Here, heterogeneous attribute labels are embedded for each of the graph nodes, and a dependency strength quantization value is assigned to each of the directed dependency edges. Based on the temporal constraints of forward inference, backward gradient calculation, and weight update during model training, the initial operator-dependent directed acyclic graph is topologically sorted and layered. After simultaneous full-link acyclicity and temporal compliance verification, the corresponding operator-dependent directed acyclic graph is generated.

3. The heterogeneous data scheduling method for model training according to claim 2, characterized in that, The step of reading the full training samples and anchoring the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph based on the full training samples includes: Based on the full training samples read, the full statistics of sample dimension, sparsity, numerical distribution and feature fluctuation amplitude are completed to generate a sample heterogeneous feature profile. Simultaneously, based on the sample heterogeneous feature profile, the sample feature transmission link corresponding to the input and output ports of the full-link operator is mapped in the operator-dependent directed acyclic graph. Based on the sample feature transmission link, the sample feature processing rules and transmission paths of each operator execution unit are identified, so as to construct the corresponding endogenous association link between the sample and the operator. Based on the execution timing constraints of the operator-dependent directed acyclic graph, similarity clustering is performed on the parameter fluctuation range of operators in the same layer according to the endogenous association links, and the parameter matching degree of cross-layer upstream and downstream operators is verified. Operator parameters with scheduling adaptation conflict risks are marked and corrected simultaneously to anchor the endogenous heterogeneous attribute parameters accordingly.

4. The heterogeneous data scheduling method for model training according to claim 1, characterized in that, The step of performing static route mapping between the minimum scheduling execution unit and the computing power topology parameters to allocate the corresponding target computing node to the minimum scheduling execution unit includes: Based on the hardware architecture, communication topology, and heterogeneous adaptation capabilities, the computing power topology parameters are divided into several heterogeneous computing power domains. Simultaneously, based on the execution timing of the operator-dependent directed acyclic graph, the smallest scheduling execution units that are continuously bound upstream and downstream are aggregated into corresponding scheduling execution groups. Extract the heterogeneous attribute aggregation features, temporal constraints, and resource requirement baselines of each scheduling execution group to construct the adaptation and matching rules between the scheduling execution group and the heterogeneous computing power domain. Simultaneously, based on the adaptation and matching rules, according to the temporal priority and attribute matching degree of the smallest scheduling execution unit, match the corresponding candidate computing nodes within the corresponding heterogeneous computing power domain. The candidate computing nodes are dynamically filtered to output the target computing node.

5. The heterogeneous data scheduling method for model training according to claim 4, characterized in that, The step of dynamically filtering the candidate computing nodes to output the target computing node includes: Based on the endogenous heterogeneous attribute parameters, the corresponding heterogeneous attribute dynamic drift trajectory generated by the minimum scheduling execution unit in the complete model iteration training cycle is simulated, and the heterogeneous attribute dynamic drift trajectory is simultaneously decomposed into the resource demand fluctuation range and time-series constraint drift threshold of the whole cycle training. Based on the resource demand fluctuation range and the time-series constraint drift threshold, a full-iteration collaborative bearing simulation is performed on each of the candidate computing nodes to output the corresponding pre-matched node combination; The pre-matched node combination is subjected to full-link compliance verification, and the target computing node is simultaneously selected according to the verification results.

6. The heterogeneous data scheduling method for model training according to claim 1, characterized in that, The step of deploying the minimum scheduling execution unit within its corresponding target computing node during model iterative training, and collecting the operator execution progress and resource usage status of each target computing node, includes: Set corresponding resource consumption floating upper limit and execution timing fault tolerance window for each target computing node to construct a corresponding node execution monitoring baseline. Simultaneously, when deploying the minimum scheduling execution unit, allocate corresponding acquisition probes to each operator execution unit in the minimum scheduling execution unit. The acquisition probe is used to obtain the actual working parameters of the corresponding operator execution unit. Based on the actual working parameters, cross-node timing alignment and association marking processing is performed on the upstream and downstream bound target computing nodes to obtain the corresponding timing acquisition dataset. The time-series acquisition dataset is subjected to feature aggregation processing to generate corresponding execution feature vectors. Simultaneously, the execution feature vectors are converted in real time to generate the operator execution progress and resource occupancy status.

7. The heterogeneous data scheduling method for model training according to claim 6, characterized in that, The step of performing real-time transformation processing on the execution feature vector to generate the operator execution progress and the resource occupancy status includes: The execution feature vector is bound to each of the acquisition probes to extract the edge weight parameters of the operator-dependent directed acyclic graph. The edge weight parameters are used as check bits and embedded into the header of the bound execution feature vector to output the corresponding target feature vector. Extract the time-series fault-tolerant window parameters of the node execution monitoring baseline, and synchronously slice the target feature vector according to the time interval of the time-series fault-tolerant window parameters to obtain several time-series feature segments; Each of the time-series feature segments is matched with the type of the endogenous heterogeneous attribute parameter. The successfully matched time-series feature segments are used as the progress data of the operator execution progress, and the unmatched time-series feature segments are used as the status data of the resource occupancy status.

8. A heterogeneous data scheduling system for model training, characterized in that, The system includes: The construction module is used to decompose the forward and backward propagation computation links of the model to be trained into a set of operator execution units with execution dependencies, so as to construct an operator-dependent directed acyclic graph, and synchronously read out the full training samples, so as to anchor the endogenous heterogeneous attribute parameters corresponding to each operator execution unit in the operator-dependent directed acyclic graph according to the full training samples. The processing module is used to aggregate operator execution units whose parameter values ​​of the endogenous heterogeneous attribute parameters are within a preset parameter threshold into operator scheduling batches according to the execution timing constraints of the operator-dependent directed acyclic graph, and simultaneously bind operator scheduling batches with upstream and downstream execution dependencies to the smallest scheduling execution unit without execution dependency conflicts. The mapping module is used to collect the computing power topology parameters of each distributed computing node and synchronously perform static route mapping between the minimum scheduling execution unit and the computing power topology parameters, so as to allocate the corresponding target computing node to the minimum scheduling execution unit. The scheduling module is used to deploy the smallest scheduling execution unit inside the corresponding target computing node during the model iterative training process, so as to collect the operator execution progress and resource occupancy status of each target computing node, and simultaneously update the endogenous heterogeneous attribute parameters of each training sample in combination with the operator execution parameters, so as to complete the scheduling of heterogeneous data.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the heterogeneous data scheduling method for model training as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the heterogeneous data scheduling method for model training as described in any one of claims 1 to 7.

Images