A heterogeneous computing power cooperative scheduling system and method for mixed precision training

Abstract

The application discloses a heterogeneous computing power cooperative scheduling system and method for mixed precision training, and belongs to the technical field of artificial intelligence calculation. The system comprises: a calculation graph analysis and operator image module, which is used for analyzing and dividing a model calculation graph and extracting operator features; a heterogeneous hardware capability sensing and matching module, which is used for managing the performance profile and real-time state of heterogeneous hardware in a cluster, and matching optimal execution hardware for each calculation partition; a data flow coordination and pipeline parallel controller, which is used for generating a global execution plan, managing cross-device data dependency and communication, and optimizing execution efficiency through communication and calculation overlap. The application solves the problem of inefficient scheduling of mixed precision training in a heterogeneous environment, realizes automatic and accurate mapping of calculation tasks to heterogeneous hardware, significantly improves training speed and reduces training cost, and improves the overall resource utilization of the cluster.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, and in particular to a heterogeneous computing power collaborative scheduling system and method for mixed-precision training. Background Technology

[0002] With the exponential growth in the size of deep learning models, from millions of parameters in the early days to trillions of parameters today, the computational resources required to train these models have become a key bottleneck restricting their development. To address this challenge, the industry generally adopts two main technological approaches:

[0003] First, mixed-precision training. This technique has become a standard practice for accelerating training and reducing GPU memory usage. Its typical approach involves using FP16 or BF16 precision for forward and backward propagation to fully utilize the acceleration capabilities of modern AI accelerators (such as GPU Tensor Cores), while retaining a portion of FP32 precision master weights for gradient updates to ensure numerical stability and model convergence. In some speed-sensitive inference layers, even lower precision such as INT8 is being used to further explore performance potential.

[0004] Second, heterogeneous computing architecture. Current computing infrastructure typically consists of a combination of various types of hardware, such as NVIDIA GPUs that excel at FP16 / BF16 matrix operations, dedicated AI chips (such as various inference cards and NPUs) with extreme energy efficiency in INT8 inference, and high-performance CPUs. These hardware components exhibit significantly differentiated performance and energy efficiency advantages across different computational precisions and operator types.

[0005] However, existing mainstream deep learning frameworks, such as PyTorch and TensorFlow, have fundamental flaws in their underlying scheduling strategies when performing mixed-precision training. These systems typically treat the entire computation graph as an indivisible whole, defaulting to execution on a single type of hardware (usually a GPU). While it is technically possible to allow users to manually assign different parts of the model to different devices, this process faces significant challenges:

[0006] The "one-size-fits-all" scheduling strategy: Existing systems lack intelligent, hardware-aware scheduling capabilities and cannot automatically identify the performance and energy efficiency differences of different operators on different hardware in the computation graph. This results in the high-efficiency computing units of dedicated AI chips being idle for most of the training time, while GPUs undertake too many computational tasks that they are not good at, their hardware expertise is not fully utilized, and overall energy efficiency is low.

[0007] Significant overhead of manual optimization: Efficiently splitting the model across different hardware requires engineers with deep hardware knowledge and model tuning experience, a process that is tedious and error-prone. For complex modern models (such as Transformer and MoE) and dynamically changing cluster environments, manual optimization is almost infeasible and lacks portability.

[0008] A significant communication bottleneck exists: splitting the computation graph among heterogeneous devices introduces substantial inter-device data transfers (such as tensor transfers between GPUs and AI chips). If scheduling strategies fail to adequately account for communication overhead, the time consumed by data transfers can completely offset or even exceed the benefits of computational acceleration.

[0009] Uneven resource utilization: The above problems often lead to some expensive hardware (such as GPUs) in the cluster being overloaded and becoming bottlenecks, while other hardware with specific advantages (such as dedicated AI chips or high-performance CPUs) are idle or underutilized, making it impossible to achieve the optimal total cost of ownership for the cluster.

[0010] Therefore, there is an urgent need in this field for a technical solution that can automatically and intelligently achieve efficient collaborative scheduling of hybrid precision training in heterogeneous computing environments. Summary of the Invention

[0011] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0012] According to a first aspect of the present invention, a heterogeneous computing power collaborative scheduling system for mixed-precision training is provided, for scheduling deep learning training tasks in a computing cluster, the computing cluster comprising at least two different types of computing hardware, the system comprising:

[0013] The computation graph analysis and operator profiling module is used to parse the computation graph of the deep learning model, divide the computation graph into multiple computation partitions, and extract multi-dimensional operator features for each computation partition.

[0014] The heterogeneous hardware capability perception and matching module is connected to the computation graph analysis and operator profiling module. It is used to manage the performance profiles and real-time status of heterogeneous computing hardware in the computing cluster, and to match the optimal computing hardware for each computing partition based on the multi-dimensional operator features.

[0015] The data flow coordination and pipeline parallel controller is connected to the computation graph analysis and operator profiling module and the heterogeneous hardware capability perception and matching module. It is used to generate a global execution plan based on the matching results of the heterogeneous hardware capability perception and matching module, manage data dependencies and communication between computing sub-sections and devices, and optimize the overall execution efficiency through communication computing overlap.

[0016] According to a second aspect of the present invention, a heterogeneous computing power collaborative scheduling method for mixed-precision training is provided, applied to a computing cluster containing at least two different types of computing hardware, the method comprising the following steps:

[0017] The steps for computation graph analysis and operator profiling are as follows: Analyze the computation graph of the deep learning model, divide the computation graph into multiple computation partitions, and extract multi-dimensional operator features for each computation partition.

[0018] Heterogeneous hardware capability perception and matching steps: Obtain the performance profiles and real-time status of heterogeneous computing hardware in the computing cluster, and match the optimal computing hardware for each computing partition based on the multi-dimensional operator features.

[0019] Data flow coordination and pipeline parallel control steps: Generate a global execution plan based on the matching results of the matching steps, manage data dependencies and communication across devices in the computation intervals, and optimize overall execution efficiency through communication computation overlap.

[0020] Compared with existing technologies, the heterogeneous computing power collaborative scheduling system for mixed-precision training provided by this invention achieves the following technical effects by constructing a hardware-aware and communication-sensitive automated scheduling framework:

[0021] Significantly improved training speed: Through the collaborative work of the heterogeneous hardware capability awareness and matching module and the data flow coordination controller, the system can intelligently and precisely schedule computational partitions of different computational precision and types (such as FP16 matrix multiplication and INT8 inference) to the most performance-advantageous hardware (such as GPU Tensor Cores and dedicated AI chips) for execution, breaking the bottleneck of traditional single-hardware execution. Real-world testing shows that compared to traditional single-GPU training solutions, the overall training time can be reduced by 25%-40%.

[0022] Training costs are effectively reduced: By fully recognizing and utilizing the high energy efficiency of dedicated AI chips at specific levels of precision (such as INT8) and prioritizing tasks to such hardware, the power consumption per unit of computing task is significantly reduced. Simultaneously, by increasing the utilization rate of inexpensive and efficient hardware, reliance on expensive general-purpose GPUs is reduced, thereby lowering the total cost of ownership (TCO) for hardware leasing and electricity by 20%-35%.

[0023] Cluster resource utilization tends to be balanced: This invention completely breaks the static scheduling mode of "GPU-centric". Through cost evaluation and global scheduling of the matching module, the load can be dynamically distributed to all available heterogeneous computing power, so that various hardware in the cluster (such as GPU, NPU, CPU) can be fully utilized according to their expertise, solving the problem of idle resources and improving the overall resource utilization rate by more than 30%.

[0024] Highly automated and easy to use: It automates the entire process from computation graph analysis and hardware matching to data flow coordination. Users do not need in-depth hardware knowledge or to perform tedious manual model splitting. They only need to submit standard training scripts to transparently complete optimal heterogeneous scheduling, which greatly reduces the threshold for using heterogeneous computing and the operation and maintenance costs.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a structural block diagram of a heterogeneous computing power collaborative scheduling system for mixed-precision training provided in an embodiment of the present invention;

[0028] Figure 2 The flowchart illustrates a heterogeneous computing power collaborative scheduling method for mixed-precision training, as provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0031] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0032] (Example 1)

[0033] One embodiment of the present invention provides a heterogeneous computing power collaborative scheduling system for mixed-precision training, which is deployed in a computing cluster containing at least two different types of computing hardware for collaborative scheduling of the execution of deep learning training tasks. Figure 1 The diagram shown is an architecture diagram of the system in this embodiment. The system mainly includes: a computation graph analysis and operator profiling module 1, a heterogeneous hardware capability perception and matching module 2, and a data flow coordination and pipeline parallel controller 3.

[0034] In this invention, the “computing cluster” refers to a collection of multiple computing nodes interconnected through an internal network. These nodes are managed uniformly and work together to complete large-scale deep learning training tasks.

[0035] The core feature of the computing cluster is:

[0036] Heterogeneity: The cluster contains at least two different types of computing hardware, such as general-purpose graphics processors, neural network processors, central processing units, or other dedicated AI accelerator cards. This architectural diversity is the physical basis for the intelligent scheduling and energy efficiency optimization achieved in this invention.

[0037] Interconnectivity: All computing nodes are interconnected via a high-speed network. This network includes, but is not limited to:

[0038] Intra-node interconnects: such as PCIe and NVLink.

[0039] Inter-node interconnection: such as InfiniBand, high-speed Ethernet (RoCEv2).

[0040] Unified Resource Pool: From the perspective of the scheduling system, all heterogeneous computing hardware in the cluster constitutes a unified, dynamically allocable computing resource pool. The system can perceive the characteristics and status of each resource in the pool and perform global optimization scheduling.

[0041] Depending on the implementation, the computing cluster can be:

[0042] Physical cluster: consists of multiple servers located in a physical cluster.

[0043] Virtual cluster: In a cloud environment, a unified resource pool is formed by logically organizing distributed computing instances through a software-defined network.

[0044] The core of the "computing cluster" described in this invention lies in the heterogeneity of hardware types and the unified management of resources, rather than the limitation of its physical deployment form.

[0045] In this invention, "computing hardware" refers to a physical processing unit in a computing cluster capable of independently executing computing tasks. Its core feature is its ability to directly execute machine instructions compiled from the unified intermediate representation and complete forward propagation, backward propagation, or weight update operations in deep learning training.

[0046] The computing hardware, based on its architecture and expertise, includes, but is not limited to, the following types:

[0047] General-purpose graphics processor: It excels in parallel floating-point operations, and has been hardware optimized for matrix multiplication and convolution calculations with FP16 / BF16 precision.

[0048] Neural network processor: A dedicated processor customized for deep learning computing, typically with extreme energy efficiency in low-precision inference computing such as INT8 / INT4.

[0049] Central Processing Unit (CPU): As a general-purpose computing unit, it is responsible for performing control flow-intensive operations, data preprocessing, and system coordination tasks.

[0050] Other dedicated accelerator cards include, but are not limited to, tensor processing units, vision processing units, and other hardware focused on specific computing domains.

[0051] The term "computing hardware" as used in this invention specifically refers to the smallest unit of computing resources that can be independently allocated computing tasks by the scheduling system and has its own independent memory address space. For example, multiple GPUs in a server are considered as multiple independent "computing hardware", while multiple streaming multiprocessors on the same GPU are not considered as separate "computing hardware".

[0052] (Computational graph analysis and operator profiling module)

[0053] In this embodiment of the invention, the computation graph analysis and operator profiling module 1 is used to parse the computation graph of the deep learning model, divide the computation graph into multiple computation partitions, and extract multi-dimensional operator features of each computation partition.

[0054] In this invention, a deep learning model specifically refers to a software entity that can be represented as a directed computation graph, which consists of interconnected operators, including but not limited to matrix multiplication, convolution, and activation functions. During the forward and backward propagation training process, this model involves calculations of various numerical precisions (such as FP32, FP16 / BF16, and INT8), and due to its large scale, it needs to be split across multiple computing devices for execution.

[0055] Specifically, the execution flow of the computational graph analysis and operator profiling module includes:

[0056] S10, Computation graph partitioning step: Based on a joint analysis of data flow dependency and operator affinity, the computation graph is automatically partitioned into multiple logically independent and schedulable computation partitions. Operator affinity is used to quantitatively characterize the overall performance gains resulting from reduced data communication or the use of hardware caching when different operators are executed on the same computing hardware.

[0057] The computation graph partitioning step in the computation graph analysis and operator profiling module aims to intelligently divide a single computation graph into multiple partitions to maximize the overall execution efficiency of the heterogeneous cluster. Its specific working principle is as follows:

[0058] S101, Building Data Flow Dependencies

[0059] Traverse the computation graph of the deep learning model to construct a data flow dependency graph. Nodes represent operators, and edges represent tensor data flows. This step clarifies the execution order and data dependencies between all operators. The core principle is: if two operators with direct data dependencies are split onto two computational hardware devices that cannot perform efficient point-to-point communication, then communication operations must be explicitly inserted at the partition boundary between them (e.g., data transfer via PCIe or NVLink). This step ensures the correctness of the computational logic after partitioning.

[0060] S102, Operator Affinity Quantization

[0061] Based on the hardware capability database in the heterogeneous hardware capability awareness and matching module, an affinity score is calculated for each pair of adjacent operators in the data stream. This affinity score is used to quantify the overall performance gain of executing them on the same computing hardware, and the gain mainly comes from two aspects:

[0062] Avoid communication overhead: Avoid the latency and bandwidth consumption caused by transmitting intermediate tensors between the two.

[0063] Hardware expertise synergy: Forming a computation sequence that can continuously utilize specific hardware core computing units (such as the Tensor Core of a GPU) to improve computational efficiency.

[0064] Specifically, the affinity score = α × avoided communication cost + β × hardware expertise synergy benefit. Here, α and β are configurable weighting coefficients used to adjust the relative importance of communication and computation in the decision-making process.

[0065] The avoided communication cost quantifies the overhead saved by placing two adjacent operators (A and B) on the same hardware. Avoided communication cost = tensor transfer volume / inter-device link bandwidth. Tensor transfer volume refers to the size of the tensor from the output of operator A to the input of operator B in memory (e.g., in MB). This can be inferred from the structure of the computation graph and the shape of the tensors. Inter-device link bandwidth refers to the measured or theoretical peak bandwidth of the data transmission path between A and B if they were assigned to different devices (e.g., bandwidth via PCIe 4.0 x16, NVLink, etc.). This data can be queried from the hardware capability database.

[0066] Hardware expertise synergy benefits quantify the computational performance gains achieved by fully utilizing the hardware's computational capabilities when a set of adjacent operators in the data stream are executed on the same computing hardware. Hardware expertise synergy benefit E = ∑ n j=1 (TP) i -TC i )×w i , among which, TP i The expected execution time of operator i on its optimal hardware type is obtained by querying the hardware capability database. TC i The expected execution time of operator i on the currently evaluated hardware can be estimated based on historical performance profile data or operator characteristics (FLOPs, memory accesses) and hardware performance models. i The importance weight of operator i within the group is determined based on its computational cost or criticality. i ranges from 1 to n, where n is the number of operators evaluated simultaneously. When E > 0, it indicates that executing the operator group on the current hardware is more efficient than distributing it across the optimal hardware for each operator. When E < 0, it indicates that executing on the current hardware would incur computational performance losses, and the operator group should be split across different hardware. In a typical scenario, n = 2, meaning a pair of operators adjacent in the data stream (e.g., operator A and operator B) are evaluated.

[0067] S103, Cutting Cost Evaluation and Edge Marking

[0068] Treat the entire computation graph as a weighted graph and evaluate the cutting cost for each edge (i.e., the connection between operators).

[0069] The calculation of the splitting cost takes into account both communication costs and performance gains. The basic principle is: splitting cost ≈ introduced communication overhead - performance gains from scheduling to better hardware.

[0070] During the cost evaluation process, edges in the computational graph are marked and their costs assigned according to the following rules:

[0071] For any edge, if the volume of the transport tensor associated with the edge is greater than the dynamic communication threshold Vc, and the affinity score of its operator pair is in the top Y% of the overall score distribution (i.e., "high affinity"), then the edge is assigned a high cutting cost and marked as "should be avoided as much as possible" in the global execution plan. The technical principle behind this rule is that such edges represent extremely high communication overhead and extremely high collaborative computation benefits. The communication latency caused by cutting them is difficult to compensate for through subsequent computation acceleration, so their connection state must be maintained first.

[0072] The preset communication threshold is determined as follows: Vc = B × T0. B is the communication bandwidth between relevant devices obtained from the hardware capability database, and T0 is a predefined upper limit of the system's tolerable communication latency. T0 can be determined in the following ways: based on the latency sensitivity of the training task (e.g., real-time training requirements); dynamically adjusted through offline analysis or online learning to adapt to different models and cluster environments; and a typical value example is provided, such as T0 = 1ms. This dynamic calculation ensures that the preset communication threshold matches the actual communication capability of the current cluster. In this embodiment of the invention, Y = 20.

[0073] During the cost assessment of the cutting process, the following rules are used to identify "potentially good cutting points" suitable for partitioning:

[0074] For any edge, if the volume of the transport tensor associated with the edge is less than one-Kth of the dynamic communication threshold, and the performance gain obtained by scheduling the operators at both ends of the edge to their respective optimal computing hardware is significantly greater than the introduced communication overhead, then the edge is given a low cutting cost and marked as a "potentially good cutting point".

[0075] The technical principle behind this rule is that such edges represent extremely low communication overhead and extremely high heterogeneous acceleration potential. The latency introduced by cutting them is minimal, and considerable performance gains can be achieved by precisely scheduling operators onto specialized hardware.

[0076] Here, K is a configurable positive constant used to define the upper limit of "low communication volume". K can be set according to the percentile of the cluster network performance, with a typical value of 10, and can also be adjusted according to the cluster network performance.

[0077] The performance gain of a single operator is equal to (T) slow-i -T fast-i ), where T slow-i T is the execution time of operator i on its slow candidate hardware. fast-iLet be the execution time of operator i on its fast target hardware. For two operators at the two ends of an edge (such as operators A and B), the theoretical maximum distributed execution benefit G is the sum of the benefits of the two operators, i.e., G = (T slow-A -T slow-A )+(T slow-B -T slow-B For an operator currently being evaluated, the hardware with the longest expected execution time from the set of all candidate hardware capable of executing the operator is defined as the "slow candidate hardware" for that operator. For an operator currently being evaluated, the hardware with the shortest expected execution time from the set of all candidate hardware capable of executing the operator is defined as the "fast target hardware" for that operator.

[0078] In this embodiment of the invention, when G > R perf ×T comm Only when this is done is the performance gain considered "significant". Among them, R... perf This is a performance gain threshold, a real number greater than 1, which can be dynamically adjusted through historical task analysis or reinforcement learning, for example, equal to 1.5 or 2. comm The communication overhead introduced for this edge is the communication time. This inequality ensures that the total computation time saved by cutting and scheduling must significantly exceed (R0). perf The communication time introduced (more than twice) is reduced to ensure that the overall benefit of this partition is positive.

[0079] S104 uses an optimization algorithm to search among all possible partitioning schemes to find a final partitioning scheme.

[0080] In this embodiment of the invention, optimization algorithms (such as dynamic programming, graph neural network prediction, or heuristic search) are used to model the computation graph partitioning problem as a constrained combinatorial optimization problem and solve for the optimal partitioning scheme.

[0081] The input to the optimization algorithm is a computational graph G(V,E) with accompanying performance profile data and cutting costs, where V is the set of operator nodes and E is the set of edges.

[0082] Decision variable: Assign a partition label L(v) to each node v∈V.

[0083] Optimization objective: Minimize the objective function TotalCost = ∑ e∈Ecut C(e), where Ecut is the set of all cut edges, i.e., edges e = (u, v) that satisfy L(u) = L(v). C(e) is the cutting cost calculated for edge e in step S103.

[0084] Constraints: The partitioning scheme must satisfy the inherent data flow dependencies of the computation graph to ensure logical correctness.

[0085] The optimization algorithm may employ (but is not limited to) any of the following strategies:

[0086] a) Dynamic Programming-Based Search Algorithm: This method is suitable for computation graphs with approximately chain-like or tree-like structures. Its core lies in maintaining a state transition table. Specifically, the algorithm performs a topological traversal starting from the source node of the computation graph. For each node v... r Record a set of states, State[v] r [L] indicates that when node v r When assigned to partition L, from the source node to v r The minimum cumulative cutting cost achievable by the subgraph. State transition is achieved by considering node v. r Its predecessor node v s Whether they are divided into the same partition (i.e., whether the edge e=(v) is cut) r v s This process involves selecting the path that minimizes the cumulative cost. Finally, by backtracking from the sink node, the partitioning scheme with the minimum global cost can be obtained.

[0087] b) Cost Prediction and Search Based on Graph Neural Networks: This algorithm first uses a pre-trained graph neural network model as input. This model takes a computational graph with node (operator performance profile) and edge (cutting cost) features as input and outputs a fast cost prediction for a large number of candidate partitioning schemes. Based on these predictions, the system then selects the top-H most promising candidate schemes and determines the final optimal partition by accurately calculating their objective function values. This method is suitable for ultra-large-scale computational graphs and can obtain high-quality approximate optimal solutions within an acceptable time.

[0088] c) Greedy heuristic search: The algorithm starts with an initial partition (e.g., the entire graph as a partition) and iteratively attempts to cut it. In each iteration, it evaluates all currently uncut edges and selects the edge with the lowest cutting cost to cut, thus splitting the original partition in two. After cutting, the algorithm updates the cutting cost of the affected edge. This process is repeated until a termination condition is met (e.g., the number of partitions reaches a maximum, or the cutting cost of all remaining edges is positive and exceeds a certain threshold). This heuristic method is efficient and easy to implement.

[0089] Finally, the partitioning scheme output by any of the above algorithms is used as the final calculation partitioning result.

[0090] S11, Performance profile construction and annotation steps: Establish a multi-dimensional performance profile for each computing partition as a specific implementation of the multi-dimensional operator features.

[0091] The multidimensional performance profile constitutes the performance profile data of the computing partition. The multidimensional performance profile includes at least: the computational accuracy requirements, computational density, and operator type combination required by the dominant operators in the computing partition, as well as the shape and data reuse characteristics of the partition output tensor.

[0092] The computational precision requirement identifies and records the primary numerical precision required or used by the dominant operators within a computation partition, such as FP32, FP16 / BF16, or INT8. Computational intensity evaluates the computational characteristics of a computation partition, characterized by the ratio of its floating-point operations (FLOPs) to memory accesses. Operator type combination identifies the types of operators contained within a computation partition, such as computationally intensive matrix multiplication or convolution operations, or memory-intensive element-level operations or control-flow operations. Partition output tensor shape refers to the dimensions and sizes of the output tensor generated by the computation partition; the tensor shape directly determines the total memory usage and the regularity of memory accesses. Regular shapes (such as large matrices) are generally more conducive to efficient parallel computation and data prefetching. The data reuse characteristic metric is a parameter used to quantify the degree to which the same data block (such as weights or intermediate activation values) is used multiple times during the computation of a computation partition. This metric is an inherent attribute of the computation partition, and its value directly and quantitatively characterizes the level of data locality during the computation of that partition. Based on the numerical range of this indicator, the computational partitions are divided into different types, and scheduling strategies are formulated accordingly:

[0093] High data reuse partitions: When the metric value exceeds a certain threshold, it is determined to be a high data reuse partition. This type of partition is common in operators such as matrix multiplication and convolution, and its characteristic is that a small amount of data is repeatedly read during the calculation. These partitions are computationally intensive and should be prioritized for scheduling to high-performance hardware (such as GPU Tensor Cores).

[0094] Low data reuse partitions: When the metric value is below a certain threshold, it is determined to be a low data reuse partition. This type of partition is common in element-level operations (such as ReLU, tensor addition), and its characteristic is that data is rarely reused after being read. These partitions are memory access intensive, and their performance is limited by memory bandwidth. They should be prioritized for scheduling to high-bandwidth or general-purpose hardware.

[0095] The system presets a configurable data reuse threshold for classifying computational partitions. The specific value of this threshold can be set and adjusted through offline analysis or online learning to adapt to different models and hardware architectures. In a typical embodiment, this threshold can be set to 0.5. That is:

[0096] When a computing partition's data reuse metric is higher than 0.5, the system classifies it as a high data reuse partition.

[0097] When a computing partition's data reuse characteristic index is lower than or equal to 0.5, the system classifies it as a low data reuse partition.

[0098] S12, intermediate representation generation step: The partitioned computation graph and its attached performance profile data are compiled together into a hardware-independent unified intermediate representation. This unified intermediate representation simultaneously encodes structural information such as operator nodes and data flow dependencies of the computation graph, as well as performance annotation information such as computational precision and density in the performance profile data, thereby forming a portable program representation containing complete scheduling semantics.

[0099] Through this step, the system provides a unified, semantically rich optimization interface for different backend hardware compilers. Each compiler only needs to parse this single intermediate representation to simultaneously obtain computational logic and performance characteristics, and then implement deep optimizations for specific hardware (such as operator fusion, memory layout transformation, etc.), without needing to care about the original framework of the frontend model or the heterogeneous topology of the cluster.

[0100] Furthermore, the computation graph analysis and operator profiling module is also used to support mixed-precision training. Specifically, during the construction of the multi-dimensional performance profile, this module identifies key transition points in the computation graph from FP32 precision tensors to FP16 / BF16 precision tensors, and marks these key transition points as candidate positions suitable for inserting loss scaling operations, providing a basis for automatically inserting precision management operators in the subsequent scheduling phase.

[0101] (Heterogeneous hardware capability perception and matching module)

[0102] In this embodiment of the invention, the heterogeneous hardware capability perception and matching module 2 is connected to the computation graph analysis and operator profiling module, which is used to manage the performance profiles and real-time status of heterogeneous computing hardware in the computing cluster, and to match the optimal computing hardware for each computing partition based on the multi-dimensional operator features.

[0103] The heterogeneous hardware capability sensing and matching module includes:

[0104] A hardware capability database is used to store performance profiles of various computing hardware components in the computing cluster. The performance profile data includes at least: peak computing power, memory bandwidth, and typical energy consumption per unit of computation for different computational precisions (such as FP32, FP16 / BF16, INT8) and specific operator types (such as matrix multiplication, two-dimensional convolution).

[0105] The dynamic load monitoring unit is used to collect and maintain the real-time status of each computing hardware unit. The real-time status includes at least: computing unit utilization, available memory capacity, and the length of the task queue to be processed.

[0106] The execution cost evaluation unit is used to evaluate the overall execution cost of each computing partition across candidate hardware and make the final scheduling decision. Its workflow is as follows:

[0107] (1) For each computing partition, based on the multi-dimensional operator characteristics of the computing partition (especially the computational accuracy requirements and operator types), the computing hardware that can efficiently execute the computing partition is determined from the hardware capability database to form a set of candidate computing hardware.

[0108] In this embodiment of the invention, the candidate computing hardware must simultaneously meet all of the following conditions:

[0109] a) Computational Precision Support: The computing hardware must support the primary numerical precision required by the dominant operator of the computation partition. This is done by matching against the "Supported Computational Precisions" list in the hardware capabilities database. For example, if the dominant precision of a computation partition is FP16, then a dedicated inference card that only supports INT8 will be excluded from the candidate set.

[0110] b) Operator Type Support: The computing hardware must provide efficient implementations of all critical operators within the computing partition. This is done by querying the "Supported Operator Types" list or operator performance data in the hardware capability database. For example, if a partition contains a custom operator, and an NPU's driver library does not contain an implementation of that operator, then that NPU will be excluded.

[0111] c) Memory Capacity Constraint: The available memory capacity of the computing hardware (reported by the dynamic load monitoring unit) must be greater than the estimated peak memory consumption of the computing partition. Peak memory consumption is estimated by the computation graph analysis module based on the size of the partition's input / output tensors, intermediate activation values, and weight parameters. For example, if a partition is estimated to require 4GB of video memory, but a GPU's currently available video memory is only 3.5GB, then that GPU will be excluded.

[0112] d) Software toolchain compatibility: The hardware must have a backend compiler and runtime environment capable of compiling and executing the unified intermediate representation.

[0113] Only hardware that simultaneously meets all of the above hard constraints will be included in the "candidate computing hardware set" and enter the subsequent cost-based optimization selection stage.

[0114] (2) Based on the real-time status collected by the dynamic load monitoring unit, the execution cost of the computing partition on each computing hardware in the candidate computing hardware set is estimated by a weighted scoring function.

[0115] In this embodiment of the invention, the weighted scoring function is used to estimate the execution cost. Its core is a multi-factor linear weighted model, which aims to simulate the estimated total completion time of a computing partition on the target hardware.

[0116] In this embodiment of the invention, the weighted scoring function simulates the estimated total completion time of a computation partition on the target hardware by quantifying computation time, queuing delay, and communication overhead. This function supports configurable optimization strategies, specifically including two operating modes:

[0117] Mode 1: Performance-First Mode

[0118] In this mode, the execution cost Cexecute aims to minimize the task completion time, and its calculation formula is as follows:

[0119] Cexecute = Tcompute + Tqueue + Tcomm; where Tcompute is the computation time, which equals the total computational load of the computation partition divided by the remaining available computing power of the hardware. The total computational load of the computation partition is estimated based on the sum of floating-point operations (FLOPs) of the operators within the partition. The remaining available computing power of the hardware = peak computing power × max(c, 1 - current utilization). The current utilization is the real-time utilization of the hardware computing unit reported by the dynamic load monitoring unit (0.0-1.0). c is the minimum protection factor, typically 0.05, to ensure that the denominator is not zero and reflects the basic parallel capability that the hardware still possesses when fully loaded. Tqueue is the queue waiting time, which equals the product of the number of tasks in the task queue and the average task execution time. Tcomm is the data communication time, which is the time required to transfer the input data required by the computation partition from its current storage location to the target hardware. Data communication time = (∑input tensor size) / inter-device link bandwidth, where ∑input tensor size refers to the sum of the volumes of all input tensors required by the computation partition in the current execution step, specifically including:

[0120] Model parameters: These are the model parameters, such as weights and biases, necessary for executing this computational partition. If the current storage location of these parameters differs from the target execution hardware, their total size must be included in the total.

[0121] Intermediate activation values: Output from the upstream compute partition, used as input for the current partition. Similarly, if their storage location differs from the target hardware, they must be included in the sum.

[0122] This sum represents the total amount of data that needs to be transferred from the source device (such as CPU memory or other computing hardware) to the target computing hardware in order to start the computing partition.

[0123] In this invention, inter-device link bandwidth specifically refers to the effective data transfer rate of the physical link traversed when data is transferred from the memory of the source computing hardware to the memory of the target computing hardware within the computing cluster. This value is a path-specific parameter based on baseline data stored in the hardware capability database and can be updated in real time by the dynamic load monitoring unit to reflect the impact of dynamic changes such as network congestion on actual throughput. This definition specifically includes the following technical connotations:

[0124] (1) Effectiveness and Real-time Performance: The bandwidth refers to the actual and sustainable data transmission rate that can be achieved under the current operating state of the system. It may be lower than the theoretical peak bandwidth of the physical link, as it is affected by factors such as bus load, protocol overhead, driver efficiency, and concurrent task contention.

[0125] (2) Path Specificity: The bandwidth value is strictly dependent on the communication path formed by the specific source device and the target computing device. The system maintains or can estimate a bandwidth matrix in its hardware capability database in real time, which records or can dynamically calculate the effective bandwidth between any two devices. Typical paths include:

[0126] Device pairs interconnected via PCIe bus or NVLink within the same compute node.

[0127] Device pairs interconnected between different computing nodes via network technologies such as InfiniBand, RoCE, or high-speed Ethernet.

[0128] (3) Data acquisition and updating: The effective bandwidth data is a key component of the hardware capability database, and its acquisition methods include:

[0129] Initialize benchmark test: When deploying the system or adding new hardware, measure and initialize this value by running a standard data transfer benchmark test program.

[0130] Runtime dynamic monitoring: During system operation, the dynamic load monitoring unit periodically queries the link status through the driver interface or performs lightweight micro-benchmark tests to detect changes such as network congestion and update the bandwidth value.

[0131] Application in communication time calculation: In the cost assessment, the "inter-device link bandwidth" used to calculate Tcomm is the effective bandwidth of the path from the data source device to the target candidate device at the current moment, determined according to the above principles.

[0132] Mode 2, Energy Efficiency Priority Mode

[0133] In this mode, the execution cost Cenergy aims to minimize energy consumption, and its calculation formula is as follows:

[0134] Cenergy = Etotal = Pd × (Tcompute + Tqueue + Tcomm), where Pd is the dynamic power of the hardware under the current load, which is obtained from the hardware capability database or real-time monitoring data. The system will select the hardware with the lowest Cenergy, which means prioritizing the computing hardware that can perform the most computations per unit of energy (i.e., the highest energy efficiency).

[0135] (3) Based on the estimated execution cost, select the computing hardware with the lowest execution cost from the candidate computing hardware set as the optimal hardware for executing the computing partition.

[0136] Finally, the execution cost evaluation unit calculates the execution cost (Cexecute or Cenergy) of all candidate hardware based on the selected working mode, and selects the computing hardware with the lowest cost as the optimal scheduling target.

[0137] In this embodiment of the invention, the execution result of the heterogeneous hardware capability awareness and matching module is manifested as the generation of a partition-hardware mapping table. This mapping table is a key data structure connecting computation graph analysis and global execution plan generation. Specifically, the partition-hardware mapping table is a logical set of mapping relationships, where each entry associates the unique identifier of a computation partition with the unique identifier of its optimal target computation hardware. Each entry not only records the mapping relationship between the partition and the hardware but may also contain metadata such as the estimated execution cost and decision timestamp, allowing the data flow coordination and pipeline parallel controller to perform more granular scheduling when generating the plan. This mapping table encapsulates all the decision results of the matching module, providing a direct and operable input basis for the subsequent controller to generate a global execution plan. By querying this table, the controller can clearly know where each computation partition should be scheduled for execution.

[0138] Furthermore, the heterogeneous hardware capability perception and matching module also includes an online learning unit for self-evolution and optimization of the system performance prediction capability. This unit operates through the following steps:

[0139] S20: Performance Data Collection

[0140] During system operation, real-world performance metrics for each computing partition on its actual execution hardware are continuously and asynchronously collected. These metrics include at least: actual execution time, actual energy consumption, and queuing delays caused by resource contention.

[0141] S21: Data-Driven Optimization

[0142] Based on the large amount of real-time performance data collected, subsequent scheduling decisions are dynamically optimized through one or more of the following methods:

[0143] Dynamically update the hardware capability database: Use statistical methods (such as moving average or exponential smoothing) to smoothly update the performance profiles (such as the execution time of specific operators) in the hardware database, making them closer to the actual performance of the hardware in the current system environment, so that subsequent scheduling decisions are based on more accurate performance data.

[0144] Training a lightweight performance prediction model: Taking the multi-dimensional performance profile of the computation partition and hardware status as input, and the actual execution time as the objective, a regression prediction model is trained online. This lightweight performance prediction model provides its output execution time estimate to the execution cost evaluation unit. The execution cost evaluation unit then corrects its initial estimate based on theoretical computing power and a simple model. Specific correction strategies include, but are not limited to, the following two modes:

[0145] Alternative approach: Directly use the output value of the performance prediction model as the final estimated execution time of the computing partition on the target hardware.

[0146] Fusion Mode: The output value of the performance prediction model is weighted and fused with the initial estimate, and the fused result is used as the final estimated execution time. The weights can be dynamically adjusted based on the confidence level of the model prediction or the historical accuracy.

[0147] By introducing online learning units, this invention enables the scheduling system to adapt to different model structures, dynamically changing cluster workloads, and performance changes brought about by hardware driver updates, effectively overcoming the limitations of fixed performance profiles, thereby continuously improving the quality of scheduling decisions and the overall operating efficiency of the cluster.

[0148] (Data flow coordination and pipelined parallel controller)

[0149] In this embodiment of the invention, the data flow coordination and pipeline parallel controller 3 is connected to the computation graph analysis and operator profiling module and the heterogeneous hardware capability perception and matching module. It is used to generate a global execution plan based on the matching result of the heterogeneous hardware capability perception and matching module (i.e., the partition-hardware mapping table), manage the data dependency and communication between computing partitions and devices, and optimize the overall execution efficiency through communication computing overlap.

[0150] In this invention, the global execution plan is a structured, executable sequence of instructions and a resource mapping scheme generated by the controller. It serves as the final carrier for transforming the abstract computation graph into a physical execution scheme that can run correctly and efficiently on specific heterogeneous hardware topologies. This plan encapsulates the following core information:

[0151] Task topology: It clarifies the execution order and dependencies of each partitioned computing unit on its matching target computing hardware.

[0152] Communication matrix: Explicitly defines all cross-device communication operations, including the data source, target, transmitted data tensor, and the communication primitives used (such as send, recv, all-reduce).

[0153] Synchronization strategy: Integrates synchronization points required for correctness and high performance, such as the token- or version-number-based synchronization mechanism.

[0154] Optimized instructions: These include pipeline parallel control instructions and asynchronous data prefetch instructions inserted to enable overlapping of communication and computation.

[0155] In this embodiment of the invention, the global execution plan is generated through the following steps:

[0156] S301, Input Integration and Initialization: The data flow coordination and pipeline parallel controller receives the partitioned computation graph from the computation graph analysis and operator profiling module, and the partition-hardware mapping table from the heterogeneous hardware capability awareness and matching module; initializes an empty, timeline-based global execution plan framework.

[0157] S302, Task Node Mapping and Sorting: Each partitioned computing area is instantiated into a specific computing task on its target computing hardware according to the mapping table. Based on the inherent data flow dependencies of the computing graph, all computing tasks are topologically sorted to determine their basic execution order, resulting in a sorted task sequence.

[0158] S303: Communication Operation Insertion: Traverse the sorted task sequence, identify all locations where consecutive computation tasks are scheduled to different computing hardware, these locations are the communication boundaries, and at each communication boundary, explicitly insert one or more pairs of communication primitives (such as Send and Recv), and define the quintuple attributes for each communication operation, the quintuple attributes including the source data tensor, source computing hardware, target computing hardware, communication primitive (such as point-to-point, set communication) and the preceding computation task it depends on.

[0159] S304, pipelined parallel integration:

[0160] Pipeline parallelism overlaps the output transfer of upstream tasks with their own computational portions, ensuring that the Recv operation of downstream tasks completes before its computation begins. This overlap is achieved in the plan by setting asynchronous execution flags and dependencies.

[0161] Data prefetching: In the plan, at the start of the current computation task, prefetch instructions for the data required by its direct successor task are inserted asynchronously.

[0162] Synchronization mechanism embedding: At key nodes such as weight updates, token request / release instructions or version number check / update instructions are inserted into the plan according to the selected strategy to manage the weight update order across heterogeneous hardware and ensure the correctness of model convergence.

[0163] In one illustrative embodiment, the synchronization mechanism is a token-passing-based synchronization mechanism, including:

[0164] (1) Maintain at least one logical token, which represents a permission to update a set of weights;

[0165] (2) Before performing a weight update operation, the computing hardware must apply for and obtain the corresponding token. Before updating the weights within its computing partition, the computing hardware must apply for and obtain the corresponding token from the token management unit.

[0166] (3) Perform weight updates during token holding. During token holding, the computing hardware has the right to perform local weight update operations.

[0167] (4) After the update operation is completed, release or pass the token. After the update operation is completed, the computing hardware must immediately release or pass the token to the next piece of hardware that needs to update the weight according to the data flow dependency.

[0168] In another illustrative embodiment, the synchronization mechanism is the version number-based synchronization mechanism, including:

[0169] (1) Maintain a global version number for the weight tensors that need to be synchronized. A typical implementation of the global version number in the system is a 64-bit unsigned integer counter. The counter is initially set to 1, and the system increments its value precisely by 1 every time the weight tensor is successfully updated, using an atomic instruction (such as compare-and-swap). This implementation provides the system with a lightweight and reliable criterion for determining the order of operations.

[0170] (2) When reading the weight tensor, the computing hardware records the global version number of the weight tensor as a reference version number, and before performing an update, verifies whether the current global version number of the weight tensor is consistent with the reference version number. When reading weights for forward propagation, the computing hardware records the version number Vread it reads. When preparing to update the weights after backpropagation, the computing hardware first checks whether the version number of the current weight is still Vread.

[0171] (3) If the current version number is the same as the reference version number, then perform the update and atomically increment the global version number.

[0172] (4) If the current version number is inconsistent with the reference version number, it means that there is a conflict with other updates. If the update fails, the update failure and retry mechanism will be automatically triggered.

[0173] 1) Failure determination and status rollback

[0174] Failure Confirmation: The controller confirms that the weight update operation failed because the data version has changed.

[0175] Gradient preservation: The computing hardware retains the currently computed gradient tensor and discards only invalid update results calculated based on outdated weight data.

[0176] Status logging: Records the number of retries and the reasons for failures, used for subsequent optimization decisions. 2) Automatic retry process

[0177] The controller initializes and manages the retry counter, then enters an automatic retry loop, which includes the following steps until the update is successful or the termination condition is met:

[0178] Step 1: Data refresh: The controller instructs the computing hardware to reread the latest data of the target weight tensor and its current global version number from storage, and update the local reference version number to the latest value.

[0179] Step 2: Gradient Reapplication:

[0180] The computing hardware directly uses the cached gradient tensor to recalculate the weight update based on the newly read weight data; this process does not require re-execution of forward and backward propagation, and only involves arithmetic operations between gradients and weights, resulting in minimal computational overhead.

[0181] This step involves only simple arithmetic operations on gradients and weights, with minimal computational overhead. Step 3: Retry Submission: The computing hardware attempts to submit the update again, performing version verification.

[0182] If the current global version number is the same as the version number read this time, then the weight update is performed and the global version number is atomically incremented, indicating that the retry was successful and the loop terminates.

[0183] If there is a discrepancy, the retry will fail and will be recorded by the controller.

[0184] 3) Loop control and termination

[0185] Loop control: After each failed retry, the controller increments the retry counter by 1. If the counter value does not exceed the preset maximum retry threshold, the controller may first perform a random backoff delay, and then jump to step one to continue the next retry.

[0186] Termination process: If the value of the retry counter exceeds the maximum retry threshold, the controller determines that it is an unrecoverable error, terminates the training task of the computing hardware, and reports the abnormal information to the system management unit.

[0187] 4) Gradient Cache Management

[0188] A lightweight gradient cache is maintained for each computation partition; the cache only stores the gradient tensors calculated in the current iteration and is automatically cleared after the weights are successfully updated. For large model scenarios, gradient compression techniques can be used to reduce cache memory usage.

[0189] The above-mentioned update failure and retry mechanism has the following advantages:

[0190] Extremely low overhead: The retry process only involves rereading the weight data and reapplying the gradient, avoiding costly repetitive computations during forward and backward propagation;

[0191] Fast recovery: Retry latency is reduced from hundreds of milliseconds (recalculation) to microseconds (data read + simple calculation);

[0192] Memory-friendly: Gradient cache has limited footprint and can be optimized by combining with existing memory management mechanisms;

[0193] Maintaining correctness: Significantly improves system responsiveness and throughput while ensuring strong data consistency.

[0194] S305: Resource Allocation and Output: Based on the planned tasks and communication, estimate and mark the resource usage of each hardware component, such as video memory and bandwidth. Compile all the above information (computation tasks, communication operations, dependencies, optimization instructions) into a unified final execution plan that can be parsed and scheduled by the underlying runtime system. This plan can be viewed as a static data flow graph, where nodes are computation / communication operations and edges are data flows and control dependencies.

[0195] Through the aforementioned automated process, the user's abstract model is transformed into a detailed "construction blueprint" that can guarantee computational correctness and maximize parallel efficiency on heterogeneous hardware topologies, thereby achieving coordinated scheduling of high performance and high resource utilization.

[0196] When generating the global execution plan, the data flow coordination and pipeline parallelism controller specifically implements the following key functions:

[0197] S31 explicitly inserts and defines data dependencies and communication operations (such as point-to-point transmission and aggregate communication) across device computation partitions in the generated global execution plan, transforming the abstract computation graph into a sequence of instructions that can be executed correctly and efficiently on heterogeneous hardware.

[0198] Explicitly refers to explicitly and directly including communication operations (such as send, recv, allreduce) as independent instructions or nodes in the generated execution plan, at the same level as computation operators. This contrasts with the implicit handling of data transfer by the driver and runtime system when running on a single device. Insertion refers to actively adding these communication instructions at compile time or runtime by the data flow coordinator and pipeline parallel controller at the boundaries of computation partitions requiring data transfer. Definition refers to specifying all attributes of each communication operation, including: source device, target computing device, transferred data tensor, dependent computation tasks, and the type of communication primitive used.

[0199] S32 employs pipelined parallelism to overlap the output data transmission operations of the upstream computing partition with the computation process of the downstream computing partition in time, thereby hiding communication latency. The data flow coordination and pipelined parallelism controller, through double buffering or streaming processing mechanisms, ensures that while the downstream computing partition is computing the current batch of data, the upstream computing partition is transmitting the next batch of data, thus hiding communication latency within the computation time.

[0200] S33, for tensors (such as model weights) that need to be shared across multiple devices, based on hardware capability data (such as memory bandwidth and network topology) and real-time load in the heterogeneous hardware capability awareness and matching module, the placement location of the primary replica of the tensor is intelligently determined to minimize overall access latency. Simultaneously, the data flow coordination and pipeline parallel controller plans the optimal data transmission path for gradient synchronization during backpropagation, for example, by constructing a gradient transfer tree to optimize the efficiency of the All-Reduce operation.

[0201] In this embodiment of the invention, the data flow coordination and pipeline parallel controller determines the placement location of the primary copy of the tensor to be shared through a cost-based optimization algorithm. The core of this algorithm is to minimize the total access cost of the tensor. The total access cost C... total For: C total =∑ m u=1 (F) u ×Lcommon u )+C hardware , of which F u This represents the frequency and criticality of access to this tensor by the u-th computing device, where u ranges from 1 to m, and m is the number of computing devices sharing this tensor. Lcommon u This represents the communication delay from the candidate placement location to the u-th computing device. Its value is equal to the tensor size divided by the inter-device link bandwidth and is affected by the real-time network load.

[0202] C hardwareThis is a hardware access characteristic correction term used to quantify the system-level benefits or costs introduced by selecting a specific hardware type. This correction term is based on the hardware's memory subsystem characteristics (such as bandwidth, latency, and type) and tensor access patterns.

[0203] For high-bandwidth memory (such as HBM2e, HBM3, bandwidth > 500 GB / s), C hardware A negative value, ranging from [-1.0, 0), represents the system-level benefit gained from choosing high-performance hardware.

[0204] For standard memory (such as GDDR6, bandwidth 100-500 GB / s), C hardware Approaching zero.

[0205] For low-speed memory (such as DDR4, bandwidth <100GB / s), C hardware A positive value, ranging from (0, 1.0], represents the additional system cost caused by hardware performance bottlenecks.

[0206] The specific values ​​are obtained by querying a predefined correction table in the hardware capability database. This predefined correction table is initialized with a set of default values ​​based on hardware specifications and benchmark test results during system initialization (e.g., -0.5 for HBM2e memory, 0.0 for GDDR6 memory, and +0.5 for DDR4 memory). During operation, the online learning unit dynamically adjusts these correction values ​​by analyzing the difference between the actual and estimated tensor access latency.

[0207] In the cost model, for tensors that require frequent access (such as model weights), placing their primary copies in "high-bandwidth memory" will significantly reduce the hardware access characteristic correction term, thereby greatly optimizing the total access cost.

[0208] The data flow coordination and pipelined parallel controller traverses all candidate computing hardware and calculates the estimated total access cost C when placing the tensor master copy on that hardware. total Ultimately, the choice was made to make C. total The candidate hardware with the smallest value is chosen as the final placement location for the primary copy of this tensor.

[0209] Taking a weight tensor as an example, it is frequently accessed during training by GPUA (responsible for forward propagation), NPUB (responsible for some computation), and GPUC (responsible for gradient updates). Through the cost model calculations described above, it is found that although GPUC does not perform all operations, it undertakes the critical synchronous update task and has a high-speed NVLink interconnect with GPUA. Therefore, placing the primary copy on GPUC minimizes the total access cost. The data flow coordination and pipeline parallel controller will automatically complete the deployment based on this result.

[0210] In this embodiment of the invention, the essence of path planning is to design an efficient communication mode for the All-Reduce operation to minimize the completion time of the synchronization operation. The controller automatically selects the optimal communication strategy by sensing the network topology and the size of the gradient tensor. Its decision logic is based on a predefined cost model: when the number of computing devices participating in synchronization is less than or equal to a preset number of devices (e.g., 4 or 8) and the gradient tensor is less than or equal to a preset gradient volume threshold (e.g., 16MB or 32MB), a simple parameter server mode can be used to reduce scheduling complexity; otherwise, when the number of computing devices participating in synchronization is greater than the preset number of devices or the gradient tensor is greater than the preset gradient volume threshold, a topology-aware tree-structured All-Reduce algorithm is preferred to maximize parallel bandwidth and ensure the scalability of the system under high load scenarios.

[0211] A basic communication strategy is the parameter server model: a central computing device is designated to collect all gradients, perform summation calculations on this device, and then distribute the results to all devices. This method is simple to implement, but it can cause the central computing device to become a communication bottleneck, has poor scalability, and is not suitable for large-scale heterogeneous clusters.

[0212] To address the aforementioned bottlenecks, this invention preferably employs a topology-aware tree-based All-Reduce algorithm. This strategy hierarchically and divides communication tasks, fully utilizing the network topology to achieve parallel communication across multiple links. The specific steps are as follows:

[0213] S3201: Topology Awareness: Acquires network topology and real-time bandwidth information among all computing devices participating in gradient synchronization, and identifies high-performance inline links (such as NVLinks within the same node) and inter-node network links (such as those connected via InfiniBand switches).

[0214] S3202: Constructing a Hierarchical Communication Tree: Based on the perceived network topology, construct one or more tree structures for communication. The principle of constructing the tree structure is to minimize the communication latency of each link in the tree and to balance the load across the links as much as possible. This process can be implemented using graph theory algorithms, for example, using devices as nodes and the reciprocal of the link bandwidth as the edge weight, to construct one or more minimum spanning trees or shortest path trees. For example, within a server, construct a subtree with the GPU with the highest NVLink bandwidth as the root node; between different servers, select the device with the highest inter-server bandwidth as the root node for cross-server connections, thus forming a hierarchical communication tree.

[0215] S3203: Execute Tree-based All-Reduce: Gradient synchronization operations are executed in two phases along the constructed communication tree.

[0216] The Reduce-Scatter phase (bottom-up reduction): Starting from the leaf nodes, each leaf node reduces its own gradient data and the gradient portions transmitted from its child nodes (e.g., by summing), and then passes the reduction result up to the parent node. This phase distributes the complete gradient tensor reduction task across all paths.

[0217] All-Gather phase (top-down broadcasting): After obtaining the complete global reduction result at the root node, this global reduction result is broadcast downwards along the tree structure to ensure that all participating devices obtain the complete global gradient.

[0218] In this way, the communication load is evenly distributed across multiple links, avoiding single-point bottlenecks, thereby achieving optimal or near-optimal communication path planning and significantly shortening gradient synchronization time.

[0219] Furthermore, the data flow coordination and pipeline parallel controller also integrates an asynchronous data prefetching mechanism, which is used to transmit the data required by the next computing partition to the cache of the target computing device in advance when the current computing partition is being executed, thereby completely hiding the data communication time within the computing time.

[0220] The workflow of the asynchronous data prefetching mechanism is as follows:

[0221] (1) Prefetching analysis and triggering

[0222] After the global execution plan is generated, the data flow coordination and pipeline parallel controller analyzes the data flow dependencies of the computation graph to determine the set of direct successor partitions for each computation partition. When any computation partition begins execution, the asynchronous data prefetching mechanism immediately and asynchronously triggers a task to prefetch data for all its direct successor partitions.

[0223] (2) Data identification and transmission

[0224] For each subsequent partition that needs to be prefetched, the system accurately identifies all input tensors that its execution depends on, including model parameters and intermediate activation values. By querying the heterogeneous hardware capability awareness and matching module, it obtains the bandwidth information of the data transmission path and initiates an independent, non-blocking DMA transfer operation in the background to send the required data directly from the source storage location to the cache of the target computing hardware.

[0225] (3) Resource and Consistency Management

[0226] The data flow coordination and pipeline parallelism controller performs global monitoring and quota management of the memory and bandwidth resources used by concurrent prefetching operations to prevent resource contention caused by excessive prefetching. To ensure data consistency, if the source copy of the prefetched data is updated before the transmission is complete, the system will automatically establish a memory barrier or invalidate the prefetching operation and trigger a re-prefetch, thereby ensuring computational correctness.

[0227] Through the asynchronous data prefetching mechanism, the data required by the downstream computing partition is highly likely to be in the local cache when it starts to execute, thereby almost completely hiding the data communication delay within the computing time, effectively eliminating device idle waiting, and significantly improving hardware utilization and overall training throughput.

[0228] Furthermore, the data flow coordination and pipeline parallel controller is also used to implement a lightweight synchronization mechanism, and the data flow coordination and pipeline parallel controller is also used to implement a synchronization mechanism in the global execution plan to ensure that, in a distributed heterogeneous environment, the weight update operations across different computing hardware meet the order and consistency requirements for model convergence.

[0229] Furthermore, the data flow coordination and pipeline parallel controller, based on the candidate positions determined by the computation graph analysis and operator profiling module, automatically inserts a loss scaling operator before the transition point when generating the global execution plan, and creates corresponding gradient scaling and recovery operations for the loss scaling operator to ensure the correctness of gradient updates.

[0230] The loss scaling operator is used to multiply the loss value from the FP32 computation region by a scaling factor during the forward propagation process, and convert the scaled loss value to FP16 / BF16 precision for use by the downstream low-precision computation partition.

[0231] The gradient scaling operation, during backpropagation, automatically multiplies the FP16 / BF16 precision gradients propagated from upstream by the same scaling factor when the gradient flows through the loss scaling operator, thus restoring the magnitude of the gradient values. The gradient restoration operation, before the final weight update, converts the scaled gradients back to FP32 precision and uses them for the main weight update, thereby ensuring the highest stability of numerical computation and the final convergence of the model.

[0232] The data flow coordination and pipeline parallel controller integrates scaling factor management logic, which can dynamically adjust the value of the scaling factor based on the dynamic range of the gradient during training (e.g., monitoring whether the gradient overflows or underflows), thereby achieving intelligent and adaptive loss scaling.

[0233] Through the aforementioned fully automated mechanism, this invention seamlessly and transparently integrates the loss scaling technique, which is crucial yet cumbersome and error-prone in mixed-precision training, into the heterogeneous scheduling process. Users can automatically enjoy the speed advantages and stability of mixed-precision training without modifying the training scripts, greatly reducing the barrier to entry and improving the usability and robustness of the entire training system.

[0234] The present invention will now be described in detail through a preferred embodiment to demonstrate its actual performance in complex heterogeneous environments.

[0235] (I) System Deployment and Heterogeneous Environment

[0236] The operating environment of this embodiment is a typical heterogeneous computing cluster, which is configured as follows:

[0237] Computing hardware: A hybrid cluster consisting of NVIDIA A100 GPUs (excelling in FP16 / BF16 matrix operations), Huawei Ascend NPUs (extremely energy efficient for FP16 and INT8 calculations), and AMD EPYC CPUs (responsible for high-precision general-purpose computing).

[0238] Software Integration: This invention is implemented as a transparent extension library of the PyTorch deep learning framework. It intercepts the user model at runtime, parses and rewrites it through the computation graph analysis and operator profiling module, without requiring any modification to the user code.

[0239] (II) Detailed Explanation of Automated Cooperative Scheduling Process

[0240] Taking a standard training script for a ViT-Huge model submitted by a user as an example, the scheduling process of this system is as follows:

[0241] 1. Computational graph analysis and intelligent partitioning stage

[0242] The computational graph analysis and operator profiling module parses the ViT-Huge model and automatically classifies it based on operator type and numerical sensitivity:

[0243] FP16 / BF16 computation partition: Contains most of the matrix multiplication and convolution intensive operations of the model to make full use of the TensorCore of modern accelerators.

[0244] FP32 computation partitioning: Identify operations in the attention mechanism that are sensitive to numerical precision, such as Softmax and LayerNorm, and classify them into FP32 partitions to ensure convergence stability.

[0245] INT8 computation partition: The classification header and other parts at the end of the model are quantized and marked as partitions that can perform INT8 quantized inference.

[0246] 2. Heterogeneous Hardware Awareness and Precise Matching Stage

[0247] The heterogeneous hardware capability perception and matching module performs precise scheduling based on the performance profiles of each partition:

[0248] FP16 / BF16 partitions are prioritized for scheduling on NVIDIA A100 GPUs and Huawei Ascend NPUs, as both can provide peak computing power at this level of precision. The matching algorithm dynamically selects specific devices based on real-time load to achieve load balancing.

[0249] FP32 partitioning: scheduled for execution on AMD EPYC CPUs. This frees up dedicated computing units in the GPU / NPU, allowing it to focus more on its strengths in high-throughput computing.

[0250] INT8 partitions are precisely and exclusively scheduled to Huawei Ascend NPUs to take advantage of their extreme energy efficiency and inference speed at INT8 precision.

[0251] (III) Data Flow Coordination and Pipeline Execution Phase

[0252] The data flow coordination and pipeline parallel controller generates a global execution plan and performs aggressive overlap optimization:

[0253] For example, once the NPU completes the computation of an FP16 partition, its output activation tensor is immediately pushed asynchronously to the CPU, and the NPU can begin processing the next batch of data without waiting.

[0254] While the CPU performs FP32 calculations, data transfer also occurs in parallel between the NPU and the GPU. This fine-grained pipelined parallelism technique significantly reduces the overhead of originally serial inter-device communication. Real-world testing shows that, in this embodiment, communication overhead is successfully controlled to within 10% of the total execution time.

[0255] (iv) Actual performance and beneficial effects

[0256] On the same ViT-Huge model and the same dataset, compared with the baseline scheme in this embodiment where all clusters use NVIDIA A100 GPUs as computing units, the present invention achieves the following significant, quantifiable beneficial effects:

[0257] Training speed is significantly improved:

[0258] Training throughput increased by 32% (based on the number of images processed per day).

[0259] The average time per training iteration was reduced by 28%.

[0260] Training costs are effectively reduced:

[0261] By precisely scheduling a large number of computing tasks to the energy-efficient NPU and CPU, and avoiding complete reliance on the high-power GPU, the overall energy consumption is reduced by 25%.

[0262] Resource utilization is trending towards equilibrium:

[0263] The utilization rate of Huawei Ascend NPU has increased from less than 15% under traditional scheduling strategies to over 75%.

[0264] It breaks through the utilization bottleneck of "GPU-centric" and significantly increases the overall computing resource output of the cluster.

[0265] This embodiment fully demonstrates that the automated collaborative scheduling system described in this invention can intelligently achieve optimal matching between mixed precision training and heterogeneous computing power, simultaneously achieving the three core goals of speeding up, reducing costs, and increasing efficiency without increasing the burden on users, and solving all the key problems mentioned in the background art.

[0266] (Example 2)

[0267] This invention provides a heterogeneous computing power collaborative scheduling method for mixed-precision training, applicable to computing clusters containing at least two different types of computing hardware, such as... Figure 2 As shown, the method is executed by the heterogeneous computing power collaborative scheduling system and includes the following steps: computation graph analysis and operator profiling step: parse the computation graph of the deep learning model, divide the computation graph into multiple computation partitions, and extract multi-dimensional operator features of each computation partition.

[0268] The computation graph analysis and operator profiling module analyzes the computation graph of the deep learning model, divides the computation graph into multiple computation partitions based on data flow dependency and operator affinity, and extracts multi-dimensional operator features for each computation partition, including computational accuracy requirements, computational density, and operator type.

[0269] Heterogeneous hardware capability perception and matching steps: Obtain the performance profiles and real-time status of heterogeneous computing hardware in the computing cluster, and match the optimal computing hardware for each computing partition based on the multi-dimensional operator features.

[0270] The heterogeneous hardware capability sensing and matching module performs the following sub-steps:

[0271] Obtain the performance profiles and real-time status of the heterogeneous computing hardware in the computing cluster.

[0272] For each computing partition, a set of candidate computing hardware capable of executing that partition is determined based on its multi-dimensional operator characteristics.

[0273] An execution cost evaluation algorithm is used to estimate the execution cost of the computing partition on each candidate hardware, and the computing hardware with the lowest cost is selected as the optimal match.

[0274] Data flow coordination and pipeline parallel control steps: Generate a global execution plan based on the matching results of the matching steps, manage data dependencies and communication across devices in the computation intervals, and optimize overall execution efficiency through communication computation overlap.

[0275] The following sub-steps are executed through the data flow coordination and pipeline parallel controller:

[0276] Based on the matching results, a global execution plan is generated, which explicitly defines the data dependencies and communication operations across device compute partitions.

[0277] Pipeline parallelism is employed to schedule the output data transmission of the upstream computing partition and the computing process of the downstream computing partition to overlap in time, thereby hiding communication latency.

[0278] For tensors that need to be shared, the location of the primary replica is determined based on hardware capabilities and real-time load, and the optimal communication path for gradient synchronization is planned.

[0279] Through the automated steps executed sequentially above, the method achieves precise matching and efficient collaboration between computing tasks and heterogeneous hardware, ultimately achieving the beneficial effects of improving training speed, reducing training costs, and balancing resource utilization.

[0280] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in this invention.

[0281] This invention also provides a computer-readable storage medium storing computer-executable instructions for performing the methods described in this invention.

[0282] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0283] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A heterogeneous computing power collaborative scheduling system for mixed-precision training, characterized in that, For scheduling deep learning training tasks in a computing cluster, the computing cluster comprising at least two different types of computing hardware, the system comprising: The computation graph analysis and operator profiling module is used to parse the computation graph of a deep learning model, divide the computation graph into multiple computation partitions, and extract multi-dimensional operator features for each computation partition. The heterogeneous hardware capability perception and matching module is connected to the computation graph analysis and operator profiling module. It is used to manage the performance profiles and real-time status of heterogeneous computing hardware in the computing cluster, and to match the optimal computing hardware for each computing partition based on the multi-dimensional operator features. The data flow coordination and pipeline parallel controller is connected to the computation graph analysis and operator profiling module and the heterogeneous hardware capability perception and matching module. It is used to generate a global execution plan based on the matching results of the heterogeneous hardware capability perception and matching module, manage data dependencies and communication between computing intervals and devices, and optimize the overall execution efficiency through communication computing overlap. The computational graph analysis and operator profiling module is specifically used for: Based on joint analysis of data flow dependency and operator affinity, the computation graph is automatically divided into multiple computation partitions, wherein the operator affinity is used to characterize the performance gains of different operators executed on the same computing hardware; A multi-dimensional performance profile is established for each computing partition as a specific implementation of the multi-dimensional operator features. The multi-dimensional performance profile constitutes the performance profile data of the computing partition. The multi-dimensional performance profile includes at least: the computational accuracy requirements, computational density, and operator type combination required by the dominant operators in the computing partition, as well as the shape and data reuse characteristics of the partition output tensor. The partitioned computation graph and its associated performance profile data are compiled together into a hardware-independent unified intermediate representation; When generating the global execution plan, the data flow coordination and pipeline parallelism controller is specifically used for: Explicitly define data dependencies and communication operations across device compute partitions; Pipeline parallelism is employed to overlap the output data transmission operations of the upstream computing partition with the computation process of the downstream computing partition in time. Based on the hardware capability data and real-time load in the heterogeneous hardware capability perception and matching module, the placement position of the primary copy of the tensor that needs to be shared among multiple devices is determined, and the data transmission path for gradient synchronization is planned. The data flow coordination and pipeline parallel controller is also used to implement a synchronization mechanism in the global execution plan to ensure that, in a distributed heterogeneous environment, weight update operations across different computing hardware meet the order and consistency requirements for model convergence.

2. The system according to claim 1, characterized in that, The heterogeneous hardware capability sensing and matching module includes: A hardware capability database is used to store performance profiles of various computing hardware components in the computing cluster. The dynamic load monitoring unit is used to collect and maintain the real-time status of each computing hardware, including computing unit utilization, available memory capacity, and the length of the task queue to be processed. Execution cost evaluation unit, used for: For each computing partition, a set of candidate computing hardware capable of executing the computing partition is determined from the hardware capability database based on the multi-dimensional operator characteristics of that computing partition. Based on the real-time status collected by the dynamic load monitoring unit, the execution cost of the computing partition on each computing hardware in the candidate computing hardware set is estimated through a weighted scoring function. Based on the estimated execution cost, the computing hardware with the lowest execution cost is selected from the candidate computing hardware set as the optimal hardware for executing the computing partition.

3. The system according to claim 1, characterized in that, The data flow coordination and pipeline parallel controller also integrates an asynchronous data prefetching mechanism, which is used to transmit the data required by the next computing partition to the cache of the target computing device in advance when the current computing partition is being executed.

4. The system according to claim 1, characterized in that, in, The synchronization mechanism is either a token-based synchronization mechanism or a version number-based synchronization mechanism; wherein, The token-based synchronization mechanism includes: Maintain at least one logical token, which represents a permission to update a set of weights; Before performing a weight update operation, the computing hardware must first apply for and obtain the corresponding token; Perform weight updates while holding the token; After the update operation is complete, release or transfer the token; The version number-based synchronization mechanism includes: Maintain a global version number for the weight tensors that need to be synchronized; When reading a weight tensor, the computing hardware records the global version number of the weight tensor as a reference version number, and before performing an update, verifies whether the current global version number of the weight tensor is consistent with the reference version number. If the current version number matches the reference version number, then perform the update and atomically increment the global version number; If the current version number is inconsistent with the reference version number, the update failure and retry mechanism will be triggered.

5. The system according to claim 2, characterized in that, The heterogeneous hardware capability perception and matching module further includes an online learning unit, which is used for: During system operation, the actual execution performance data of each computing partition on different computing hardware is continuously collected; Based on the collected actual execution performance data, perform at least one of the following operations: Dynamically update the performance profiles in the hardware capability database; A performance prediction model is constructed and updated. This performance prediction model takes the multi-dimensional performance profile and hardware status of the computing partition as input, outputs an estimate of the execution time of the computing partition, and provides this estimate to the execution cost evaluation unit to correct the initial estimate of the execution cost evaluation unit based on theoretical computing power.

6. The system according to claim 1, characterized in that, The computational graph analysis and operator profiling module is also used for: When establishing the multidimensional performance profile, the transition point from FP32 precision tensor to FP16 / BF16 precision tensor in the computation graph is identified, and the transition point is marked as a candidate position suitable for the insertion loss scaling operation. The data flow coordination and pipeline parallel controller automatically inserts a loss scaling operator before the transition point when generating the global execution plan, based on the candidate position, and creates corresponding gradient scaling and recovery operations for the loss scaling operator.

7. A heterogeneous computing power collaborative scheduling method for mixed-precision training, applied to a computing cluster containing at least two different types of computing hardware, characterized in that, The method is executed by the heterogeneous computing power collaborative scheduling system according to any one of claims 1 to 6, and the method includes the following steps: The steps for computation graph analysis and operator profiling are as follows: Analyze the computation graph of the deep learning model, divide the computation graph into multiple computation partitions, and extract multi-dimensional operator features for each computation partition; Heterogeneous hardware capability perception and matching steps: Obtain the performance profiles and real-time status of heterogeneous computing hardware in the computing cluster, and match the optimal computing hardware for each computing partition based on the multi-dimensional operator features. Data flow coordination and pipeline parallel control steps: Generate a global execution plan based on the matching results of the matching steps, manage data dependencies and communication across devices in the computation intervals, and optimize overall execution efficiency through communication computation overlap.

Images