Block granularity graph scheduling method that guarantees hardware efficiency conservation

Abstract

This invention discloses a block-granular graph scheduling method that ensures hardware efficiency conservation, comprising: acquiring input computation tasks and hardware capability information; selecting the optimal standard block size and partitioning the matrix; identifying block-level sparse structure; constructing a block-level bipartite graph; uniformly modeling computation, communication, DMA, prefetching, and other tasks as DAG nodes and labeling dependent edges; performing distributed graph partitioning based on intermediate nodes; selecting the maximum parallel set without data dependencies and resource conflicts in each time slice as the maximum anti-chain with resource constraints; asynchronously launching tasks to independent execution streams according to resource type, inserting minimum event synchronization only when strong rigid data dependencies exist; calling the hardware's native computation core to perform multiply-add operations within the block, without executing graph traversal logic within the block; and accumulating and aggregating local results from different partitions. This invention achieves lossless compatibility between graph scheduling capabilities and hardware peak efficiency by decoupling graph blocks, ensuring that graph scheduling does not intrude into internal block computation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence computing technology, specifically relating to a block-granular graph scheduling method that ensures hardware efficiency conservation. Background Technology

[0002] As the scale of large language models, multimodal models, and embodied intelligence systems increases, matrix multiplication, attention computation, and expert network computation become the main sources of computational power consumption. Existing technologies typically employ the following solutions.

[0003] The first category is single-operator optimization schemes based on high-performance mathematical libraries, such as cuBLAS, CUTLASS, and TensorRT. These schemes maximize single-operator throughput by using mechanisms such as fixed-shape matrix blocks, memory alignment, and register reuse. However, the optimization boundary remains at the single-operator or local fusion level, making it difficult to perceive upper-level task DAGs, cross-operator dependencies, distributed communication, and global scheduling objectives.

[0004] The second category is operator scheduling schemes based on computation graphs of deep learning frameworks, such as PyTorch and TensorFlow. These frameworks represent the model as a static or dynamic graph and execute it in topological order. They can express a certain degree of operator dependencies, but they usually take the model layer or operator as the core abstraction and lack a unified view from communication transmission, memory handling, hardware queues to accelerator instructions. They can guarantee the correctness of dependencies but cannot guarantee that hardware resources are fully utilized in each time slice.

[0005] The third category is distributed training solutions based on data parallelism, model parallelism, and pipeline parallelism, such as ZeRO optimization and parameter splitting. These solutions achieve cross-device synchronization through communication primitives such as AllReduce and AllGather, which can improve throughput within a certain scale. However, when the cluster expands to hundreds or even thousands of devices, communication volume, synchronization waiting, and load imbalance will quickly become bottlenecks.

[0006] The fourth category is distributed task execution schemes based on graph scheduling or graph computation. These schemes represent tasks and dependencies using nodes and edges, and achieve distributed execution through graph partitioning, effectively expressing dependencies and parallel relationships. However, if directly applied to matrix elements or irregular, small-granular tasks, they can easily disrupt the continuity, regularity, and address alignment of matrix blocks, preventing hardware-native tensor units such as Tensor Cores and SIMD from operating stably at peak performance.

[0007] The fifth category is acceleration schemes based on sparse matrices, such as CSR, CSC, and COO formats. These schemes can reduce the theoretical computational load, but on AI acceleration chips, irregular index jumps, non-contiguous memory access, and unbalanced loads often offset the benefits of sparsity.

[0008] The core flaw in existing technologies lies in the lack of a unified and lossless mapping mechanism between graph scheduling, distributed partitioning, matrix operations, communication pipelines, and peak hardware execution. Specifically: graph scheduling treats matrix elements as nodes, leading to data block fragmentation and address misalignment; distributed graph partitioning prioritizes load balancing without considering the integrity of optimal hardware blocks, resulting in block tearing; matrix partitioning is based solely on algebraic division, lacking a unified DAG representation and hindering joint optimization; computation and communication are artificially serialized, preventing full utilization of hardware resources within the same time slice; and sparse matrices, with element-level scheduling, cannot be aggregated into standard dense blocks.

[0009] Therefore, there is an urgent need for a technical solution that can solve the above problems. Summary of the Invention

[0010] This invention provides a block-granularity graph scheduling method that guarantees hardware efficiency conservation to solve the aforementioned technical problems. Specifically, the technical solution is as follows: A block-granularity graph scheduling method that guarantees hardware efficiency conservation includes the following steps: Obtain input computing task and hardware capability information, including hardware model, number of computing units, native matrix block size, memory alignment requirements, and available execution flow information; Based on the hardware capability information, select the optimal standard block size and divide the input matrix into one or more optimal standard blocks that satisfy address alignment constraints and have continuous internal elements. Identify block-level sparse structures, remove all-zero blocks from the graph structure, and preserve non-zero blocks as regular dense blocks; Map the matrix block rows to left nodes, the common dimension blocks to middle nodes, and the matrix block columns to right nodes to construct a block-level bipartite graph. Map the matrix blocks to graph edges, and each second-order path corresponds to a block-level multiply-accumulate computation task. Block-level computing tasks, data loading tasks, communication tasks, DMA copy tasks, prefetch tasks, and write-back tasks are uniformly modeled as DAG nodes, and dependency edges and resource vectors are labeled. Distributed graph cutting is performed based on intermediate nodes or common dimension blocks, distributing block-level path tasks to different devices without tearing any hardware-optimal standard blocks. In each time slice, the largest parallel set without data dependencies and resource conflicts is selected from the set of ready nodes as the resource-constrained maximum antichain; The tasks in the maximum anti-chain are asynchronously launched to independent execution streams according to resource type, and minimal event synchronization is inserted only when there is a strong rigid data dependency. The native hardware computing core is invoked to perform multiplication and addition operations within the block, but graph traversal logic and irregular index jumps are not executed within the block. Accumulate and aggregate local results from different parts of the same output block.

[0011] Furthermore, the selection of the optimal standard block size in hardware includes: prioritizing the block size supported by the hardware native tensor unit; ensuring that the block start address satisfies the hardware alignment boundary; if the matrix size cannot be divided by the block size, then filling the boundary block or selecting a smaller standard block; if a sparse structure exists, then prioritizing the block size that can retain dense non-zero blocks.

[0012] Furthermore, the construction of the block-level bipartite graph includes: mapping the block rows of matrix A to left-hand nodes U, mapping the common-dimensional blocks to middle-hand nodes K, and mapping the block columns of matrix B to right-hand nodes V; mapping the non-zero blocks of A to edges from U to K, and mapping the non-zero blocks of B to edges from K to V; and generating a block-level computation task for each existing second-order path, the task taking a complete block as input and outputting a local result block.

[0013] Furthermore, the unified modeling of DAG nodes includes: block data loading nodes, block-level GEMM computing nodes, local accumulation nodes, cross-device communication nodes, DMA copy nodes, result reduction nodes, result write-back nodes, prefetch nodes, and cache residency and release nodes; DAG edges represent strong rigid data dependencies, weak temporal process dependencies, resource mutual exclusion dependencies, data version dependencies, and address lifecycle dependencies.

[0014] Furthermore, the distributed graph partitioning is performed while satisfying the following constraints: partitioning is performed only on intermediate nodes or the set of path tasks centered on intermediate nodes; no hardware-optimal standard blocks are partitioned; each device performs a complete block GEMM; cross-device communication occurs only during the block input routing and local result aggregation stages; and deterministic aggregation is performed on multiple partition results of the same output block.

[0015] Furthermore, the selection of the maximum anti-chain of resource constraints includes: maintaining a set of ready tasks in the current time slice; for any two tasks, if they do not have direct or indirect dependencies in the DAG, do not conflict in read and write addresses, do not exceed the hardware resource limit in the sum of resource vectors, and do not compromise block integrity and data version consistency, they can be placed in the same cross-section.

[0016] Furthermore, after selecting the maximum antichain, hardware affinity re-aggregation is performed: if the block-level tasks are too fine-grained, multiple block-level tasks with the same device, the same precision, the same block size, adjacent memory addresses and no strong dependency conflicts are re-aggregated into hardware-friendly batch execution units, and the batch tasks still maintain the independence, integrity, continuity and alignment within each block.

[0017] Furthermore, the asynchronous transmission according to resource type includes: transmitting computing tasks to a dedicated computing stream to perform tensor unit block computation; transmitting DMA copy tasks to a DMA copy stream to perform memory transfer; transmitting communication tasks to a network communication stream to perform RDMA or NVLink transmission; transmitting prefetch tasks to a prefetch stream to preload the data required for the next time slice; and transmitting result reduction tasks to a result reduction stream to complete local accumulation and cross-card reduction.

[0018] Furthermore, the invocation of the hardware native computing core to perform multiplication and addition operations within the block satisfies the following: no graph traversal logic is executed within the block; no irregular index jumps are executed within the block; the original data layout is not changed within the block; elements are not rearranged due to graph fragmentation within the block; hardware native matrix multiplication or vectorized instructions are invoked within the block; and the output local result blocks remain continuous, aligned, and aggregateable.

[0019] Furthermore, the feedback adjustment of the collection and execution indicators includes: collecting Tensor Core utilization, SIMD instruction issuance rate, cache hit rate, bandwidth utilization, communication hiding rate, scheduling overhead, and queue blocking status; if a decrease in hardware utilization is detected, the block size, re-aggregation granularity, fragmentation strategy, or issuance strategy is adjusted, rather than splitting the internal structure of the optimal standard block of the hardware.

[0020] The advantage of this invention lies in the block-granular graph scheduling method that guarantees hardware efficiency conservation. Through a graph block decoupling mechanism, the graph structure is limited to a scheduling structure responsible for dependency management and distributed routing, while the matrix block is limited to a computation structure responsible for native hardware execution. This ensures that graph scheduling does not intrude on the numerical computation process within the block, and WMMA, MMA, Tensor Core, or AI Core instructions can still be directly called within the block. Thus, the execution efficiency of the native hardware tensor computation block is not reduced before and after graph scheduling and distributed partitioning, achieving lossless compatibility between graph scheduling capabilities and peak hardware efficiency.

[0021] The advantage of this invention also lies in the block-granular graph scheduling method that guarantees hardware efficiency conservation. Using intermediate node K or common dimension blocks as the primary sharding basis, it only moves or assigns complete blocks and complete path tasks, avoiding cross-device splitting within blocks. This ensures that distributed graph sharding does not tear apart matrix blocks, thereby reducing additional communication, copying, and reassembly overhead. Cross-device communication only occurs during block input routing and local result aggregation stages; intra-block computation does not generate cross-device communication, ensuring that distributed expansion and peak hardware execution occur simultaneously.

[0022] The advantages of this invention also lie in the block-granular graph scheduling method that guarantees hardware efficiency conservation. This method issues computation tasks, DMA copy tasks, communication tasks, prefetching tasks, and result reduction tasks to independent execution streams. All issuance operations are asynchronous and lock-free, with minimal event synchronization only inserted when strong rigid data dependencies exist. This mechanism ensures that hardware resources such as computing units, DMA controllers, and RDMA network cards are fully utilized within the same time slice, breaking the bottleneck of serial execution of computation and communication in traditional frameworks and significantly improving the overall system throughput. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the block-granularity graph scheduling method that guarantees hardware efficiency conservation according to this application. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0026] In the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection" and "linkage" should be interpreted broadly, and can refer to mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc., are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change.

[0027] like Figure 1The diagram illustrates a block-granular graph scheduling method for ensuring hardware efficiency conservation, comprising the following steps: S1: Obtain input computation tasks and hardware capability information, including hardware model, number of computation units, native matrix block size, memory alignment requirements, and available execution flow information. S2: Based on the hardware capability information, select the optimal standard block size and divide the input matrix into one or more hardware-optimal standard blocks that satisfy address alignment constraints and have contiguous internal elements. S3: Identify the block-level sparse structure, delete all-zero blocks from the graph structure, and maintain non-zero blocks as regular dense blocks. S4: Map matrix block rows to left-hand nodes, common dimension blocks to middle-hand nodes, and matrix block columns to right-hand nodes, constructing a block-level bipartite graph, mapping matrix blocks to graph edges, with each second-order path corresponding to a block-level multiply-accumulate computation task. S5: Unify the block-level computation tasks, data loading tasks, communication tasks, DMA copy tasks, prefetch tasks, and write-back tasks into DAG nodes, and label dependency edges and resource vectors. S6: Perform distributed graph partitioning based on intermediate nodes or common dimension blocks, allocating block-level path tasks to different devices without tearing apart any hardware-optimal standard block. S7: In each time slice, select the largest parallel set without data dependencies and resource conflicts from the ready node set as the resource-constrained maximum anti-chain. S8: Asynchronously launch tasks in the maximum anti-chain to independent execution streams according to resource type, inserting minimal event synchronization only when strong rigid data dependencies exist. S9: Call the hardware's native computing core to perform multiply-add operations within the block; graph traversal logic and irregular index jumps are not executed within the block. S10: Accumulate and aggregate local results from different partitions for the same output block. This invention employs a graph block decoupling mechanism, limiting the graph structure to a scheduling structure responsible for dependency management and distributed routing, and the matrix block to a computation structure responsible for native hardware execution. This ensures that graph scheduling does not intrude on the numerical computation process within the block, while allowing direct invocation of WMMA, MMA, Tensor Core, or AI Core instructions within the block. Consequently, the execution efficiency of the native hardware tensor computation block remains unchanged before and after graph scheduling and distributed partitioning, achieving lossless compatibility between graph scheduling capabilities and peak hardware efficiency. The following provides a detailed explanation of the above steps.

[0028] For step S1: Obtain input computation task and hardware capability information. Hardware capability information includes hardware model, number of computation units, native matrix block size, memory alignment requirements, and available execution flow information.

[0029] Specifically, the system receives matrix computation tasks from model executors, AI compilers, or deep learning frameworks, such as GEMM, Attention, MLP, or batch matrix multiplication.

[0030] Let the input matrix be: A ∈ R (M×K) ; B ∈ R (K×N) ; The output matrix is ​​C = A×B, C ∈ R. (M×N) .

[0031] Simultaneously, the hardware capability table is read, including hardware model, number of computation units, tensor unit type, supported precision type, native matrix block size, memory alignment requirements, cache level and cache line size, DMA and NVLink / RDMA resource information, available computation streams, and event synchronization mechanisms.

[0032] For step S2: Based on the hardware capability information, select the optimal standard block size and divide the input matrix into one or more optimal standard blocks that satisfy the address alignment constraints and have continuous internal elements.

[0033] Based on hardware capabilities, prioritize block sizes supported by the native Tensor Core or AI Core, such as 16×16, 32×32, 64×64, 128×128, or 256×256. Ensure that the block start address meets hardware alignment boundaries of 16B, 32B, 64B, or 128B. Ensure the block size maximizes register reuse and shared memory reuse. If the matrix size is not divisible by the block size, pad the boundary blocks or select smaller standard blocks. If a sparse structure exists, prioritize block sizes that preserve dense non-zero blocks. Each segmented block maintains a contiguous memory layout; no graph traversal logic exists within the block, allowing direct execution by native hardware tensor instructions.

[0034] In the embodiments of this application, selecting the optimal standard block size in hardware includes: prioritizing block sizes supported by the hardware's native tensor units; ensuring that the block start address satisfies the hardware alignment boundary; if the matrix size is not divisible by the block size, then padding the boundary blocks or selecting a smaller standard block; and if a sparse structure exists, prioritizing block sizes that can preserve dense non-zero blocks.

[0035] Specifically, based on hardware capability detection results, the system prioritizes block sizes supported by the hardware's native tensor units, such as 16×16×16 or 32×32×32 supported by CUDA Tensor Cores, or 64×64×64 supported by Ascend AI Cores. Ensuring that the block start address meets the hardware alignment boundary is a prerequisite for ensuring that Tensor Cores or SIMD instructions can be issued in regular mode. When the matrix size is not divisible by the block size, padding the boundary blocks can avoid hardware efficiency degradation caused by irregular tail blocks, or a smaller standard block can be selected to fit the boundary size. When a sparse structure exists, the system prioritizes block sizes that can preserve dense non-zero blocks, allowing the non-zero regions of the sparse matrix to still be executed by hardware on standard dense blocks, avoiding index jumps and branch judgments caused by element-level sparsity. By refining the block size selection rules, the system can dynamically select the optimal block size based on the native characteristics of the hardware and the actual shape of the matrix, avoiding hardware efficiency loss caused by improper handling of boundary blocks. At the same time, it can ensure that the non-zero regions of the sparse matrix can still utilize hardware tensor units in a regular dense block manner, balancing versatility and peak hardware execution efficiency.

[0036] For step S3: Identify block-level sparse structures, remove all-zero blocks from the graph structure, and keep non-zero blocks as regular dense blocks.

[0037] Sparsity is assessed on a block-by-block basis. If all elements in a block are zero, the corresponding graph edges are deleted, and no computation task is generated. If the block contains non-zero elements but the density is insufficient, it can be compressed and reconstructed into a hardware-acceptable dense block before execution. If the block is a structured sparse block, a hardware-supported sparse kernel is invoked. If the block is a regular non-zero block, it is executed as a dense block. This step allows sparsity to reduce computational load.

[0038] For step S4: Map the matrix block rows to left nodes, the common dimension blocks to middle nodes, and the matrix block columns to right nodes to construct a block-level bipartite graph. Map the matrix blocks to graph edges, and each second-order path corresponds to a block-level multiply-accumulate computation task.

[0039] In the embodiments of this application, constructing a block-level bipartite graph includes: mapping the block rows of matrix A to left-hand nodes U, mapping common-dimensional blocks to middle-hand nodes K, and mapping the block columns of matrix B to right-hand nodes V. Non-zero blocks of A are mapped to edges from U to K, and non-zero blocks of B are mapped to edges from K to V. For each existing second-order path ui→kt→vj, a block-level computation task is generated. The input of this task is the complete blocks A[i,t] and B[t,j], and the output is a local result block.

[0040] Specifically, the blocks and rows of matrix A are mapped to left-hand nodes U, the common-dimensional blocks are mapped to middle-hand nodes K, and the blocks and columns of matrix B are mapped to right-hand nodes V, forming a three-layer block-level bipartite graph structure. Non-zero blocks of A are mapped to edges from U to K, and non-zero blocks of B are mapped to edges from K to V, with each edge carrying a complete matrix block as its attribute. For each existing second-order path ui→kt→vj, a block-level computation task is generated. This task takes complete blocks A[i,t] and B[t,j] as input and outputs a partial result block Cpartial[i,j,t]. If A[i,t] or B[t,j] is a block of all zeros, the corresponding path does not exist, and no computation task is generated.

[0041] The key to this mapping lies in using hardware-optimal standard blocks, rather than individual elements, as the smallest mapping unit. This allows the graph structure to only handle task scheduling and dependency management, while maintaining the integrity, continuity, and alignment within each block. In this step, by mapping matrix multiplication to a three-layer block-level bipartite graph, the common dimension K is transformed into a set of shardable and routable graph nodes. This provides a natural basis for subsequent distributed graph partitioning, while ensuring that each computational task uses complete hardware blocks as input and output, thus preventing the graph mapping from disrupting the continuity and regularity of the matrix blocks.

[0042] For step S5: Model block-level computation tasks, data loading tasks, communication tasks, DMA copy tasks, prefetch tasks, and write-back tasks as DAG nodes and label dependency edges and resource vectors.

[0043] In the embodiments of this application, the unified modeling of DAG nodes includes: block data loading nodes, block-level GEMM computation nodes, local accumulation nodes, cross-device communication nodes, DMA copy nodes, result reduction nodes, result write-back nodes, prefetch nodes, and cache residency and release nodes. DAG edges represent strong rigid data dependencies, weak temporal process dependencies, resource mutual exclusion dependencies, data version dependencies, and address lifecycle dependencies. Among them, weak dependencies can be relaxed or eliminated by the scheduler, while strong rigid data dependencies cannot be broken.

[0044] This step defines the node and dependency edge types included in the unified DAG modeling. Node types include block data loading, block-level GEMM computation, local accumulation, cross-device communication, DMA copying, result reduction, result write-back, prefetching, cache residency and release, covering the entire lifecycle of computation, communication, data movement, and memory management. Dependency edge types include strong rigid data dependencies (undeniable real data flows), weak temporal process dependencies (relaxable pseudo-dependencies), resource mutual exclusion dependencies (the same hardware resource cannot be occupied simultaneously), data version dependencies (multiple versions of read / write order at the same address), and address lifecycle dependencies (matching memory allocation / release with data lifetime). By distinguishing between strong and weak dependencies, the scheduler can maximize parallelism while ensuring correctness. By modeling computation, communication, DMA, I / O, and memory management as unified DAG nodes and distinguishing between strong and weak dependency types, the scheduler can relax unnecessary temporal constraints while ensuring data correctness, thereby launching a larger set of parallel tasks in each time slice and improving hardware resource utilization.

[0045] For step S6: Perform distributed graph cutting based on intermediate nodes or common dimension blocks, distribute block-level path tasks to different devices, and do not tear any hardware-optimal standard blocks.

[0046] In the embodiments of this application, distributed graph partitioning satisfies the following constraints: partitioning is performed only on intermediate nodes or the set of path tasks centered on intermediate nodes. No hardware-optimal standard block is partitioned. Each device performs a complete block GEMM. Cross-device communication occurs only during the block input routing and local result aggregation stages. Deterministic aggregation is performed on multiple partitions of the same output block.

[0047] The sharding object is limited to the intermediate node K or the set of path tasks centered on K, rather than directly cutting the matrix block itself. Constraints such as not cutting block A or B, not cutting any hardware standard block, and each device executing a complete block GEMM ensure that distributed expansion does not come at the expense of hardware execution efficiency. Cross-device communication only occurs during block input routing and local result aggregation, meaning that intra-block computation does not generate cross-device communication; data exchange only occurs during task scheduling and result aggregation. Deterministic aggregation of multiple sharding results for the same output block guarantees the numerical correctness and repeatability of distributed computation. By using the intermediate node K as the sharding basis and setting block integrity constraints, distributed graph cutting only moves or assigns complete blocks and complete path tasks, without cross-device splitting within blocks, thereby reducing additional communication, copying, and reassembly overhead and ensuring that distributed expansion and peak hardware execution occur simultaneously.

[0048] For step S7: In each time slice, select the largest parallel set of nodes with no data dependencies and no resource conflicts from the set of ready nodes as the resource-constrained maximum antichain.

[0049] Specifically, in each time slice, the system maintains a set of ready tasks. For any two tasks, if they have no direct or indirect dependencies in the DAG, their read / write addresses do not conflict, the sum of their resource vectors does not exceed the hardware resource limit, and they do not compromise block integrity or data version consistency, they can be placed in the same cross-section. The scheduler selects the largest parallel set that satisfies the above conditions from the ready set as the set of tasks to be launched at once in the current time slice.

[0050] In the embodiments of this application, selecting the maximum resource constraint anti-chain includes: maintaining a set of ready tasks in the current time slice; for any two tasks, if they have no direct or indirect dependencies in the DAG, no read / write address conflicts, the sum of resource vectors does not exceed the hardware resource limit, and they do not compromise block integrity or data version consistency, they can be placed in the same cross-section. A set of ready tasks is maintained in each time slice, consisting of all tasks whose prerequisites are satisfied. Selection criteria include: no direct or indirect dependencies in the DAG (ensuring execution correctness), no read / write address conflicts (avoiding data races), the sum of resource vectors does not exceed the hardware resource limit (avoiding resource overload), no compromise of block integrity (ensuring hardware execution efficiency), and data version consistency (ensuring computational correctness). Tasks that meet the above conditions can be placed in the same cross-section for parallel execution, and the scheduler selects the largest such set. By selecting the largest parallel set without dependency conflicts and resource conflicts from the ready tasks, the computing unit, communication unit, and data transport unit can run simultaneously within each time slice, achieving pipeline overlap between computation and communication, and reducing waiting and synchronization overhead.

[0051] For step S8: Asynchronously launch the tasks in the maximum antichain to independent execution streams according to resource type, and insert minimal event synchronization only when there is a strong rigid data dependency.

[0052] Tasks in the maximum anti-chain are launched to independent execution streams based on resource type: the computation-dedicated stream performs TensorCore or AI Core block computations; the DMA copy stream performs H2D, D2H, and D2D memory transfers; the network communication stream performs RDMA, NVLink, or PCIe transfers; the prefetch stream preloads the data needed for the next time slice; the result reduction stream performs local accumulation and cross-card reduction; and the CPU control stream performs lightweight scheduling. All launches are asynchronous and lock-free, with minimal event synchronization only inserted when there are strong rigid data dependencies.

[0053] In the embodiments of this application, asynchronous deployment according to resource type includes: deploying computation tasks to a dedicated computation stream to perform tensor unit block computation; deploying DMA copy tasks to a DMA copy stream to perform memory transfer; deploying communication tasks to a network communication stream to perform RDMA or NVLink transfer; deploying prefetch tasks to a prefetch stream to preload data required for the next time slice; and deploying result reduction tasks to a result reduction stream to perform local accumulation and cross-card reduction. By deploying tasks of different resource types to independent execution streams, hardware resources such as computing units, DMA controllers, RDMA network cards, and CPU preprocessing are fully utilized in the same time slice.

[0054] In the embodiments of this application, after selecting the maximum antichain, hardware affinity re-aggregation is performed: if the block-level tasks are too fine, multiple block-level tasks with the same device, the same precision, the same block size, adjacent memory addresses and no strong dependency conflicts are re-aggregated into hardware-friendly batch execution units, and the batch tasks still maintain the independence, integrity, continuity and alignment of each block.

[0055] Specifically, after selecting the maximum anti-chain, if the generated block-level tasks are too fine-grained, leading to increased scheduling overhead or decreased hardware launch efficiency, the system can re-aggregate multiple block-level tasks into batch execution units. Re-aggregation conditions include: same device, same precision, same block size, adjacent or same batch memory addresses, same compute core type, no strong dependency conflicts, and no crossing of hardware block boundaries. The re-aggregated batch tasks can be mapped to batched GEMM, grouped GEMM, or fused kernels, while each block remains independent, complete, continuous, and aligned. Through hardware affinity re-aggregation, a hardware-friendly batch execution mode is restored when scheduling granularity is too fine-grained, reducing kernel startup and scheduling overhead, while maintaining the native hardware execution characteristics within each block, achieving a balance between scheduling flexibility and hardware launch efficiency.

[0056] For step S9: Call the hardware native computing core to perform multiplication and addition operations within the block, without executing graph traversal logic or irregular index jumps within the block.

[0057] In the embodiments of this application, the invocation of the hardware native computing core to perform multiply-add operations within a block satisfies the following: no graph traversal logic is executed within the block; no irregular index jumps are executed within the block; the original data layout is not changed within the block; elements are not rearranged due to graph fragmentation within the block; hardware native matrix multiplication or vectorized instructions are invoked within the block; and the output local result blocks remain continuous, aligned, and aggregateable. This ensures that graph scheduling logic does not enter the device-side computing core, and that hardware native execution efficiency is maintained within the block. The absence of graph traversal logic and irregular index jumps within the block ensures continuous memory access within the block; the absence of changes to the original data layout and rearrangement of elements due to graph fragmentation guarantees the continuity and alignment of data addresses within the block; the invocation of hardware native matrix multiplication or vectorized instructions, such as GEMM, MMA, WMMA, or AMX instructions; and the maintenance of continuous, aligned, and aggregateable output local result blocks provide a hardware-friendly data layout for subsequent result aggregation. These constraints collectively ensure strict decoupling between graph scheduling logic and in-block computation logic. By clearly defining six constraints for intra-block execution, we ensure that graph scheduling logic never enters the device-side computing core, thus maintaining the optimal execution mode of the hardware's native tensor units and avoiding performance degradation caused by fragmentation, index jumps, and irregular memory accesses introduced by graph scheduling.

[0058] For step S10: Accumulate and aggregate the local results from different fragments of the same output block.

[0059] For the same output block, local results from different K-shards are accumulated. If the K-shards are distributed across multiple devices, a strategy of local aggregation first, cross-device aggregation later, or centralized aggregation on the target device is adopted. After aggregation, the output block is written back, and Tensor Core utilization, SIMD instruction issuance rate, cache hit rate, bandwidth utilization, communication hiding rate, scheduling overhead, and queue blocking status are collected to adjust subsequent block sizes, sharding strategies, and issuance strategies in reverse.

[0060] In the embodiments of this application, the collection and execution metric feedback adjustment includes: collecting Tensor Core utilization, SIMD instruction issue rate, cache hit rate, bandwidth utilization, communication hiding rate, scheduling overhead, and queue blocking status. If a decrease in hardware utilization is detected, the block size, re-aggregation granularity, fragmentation strategy, or issue strategy is adjusted, rather than splitting the internal structure of the optimal standard block of the hardware.

[0061] This step further defines the specific metrics and adjustment strategies for the feedback loop. The metrics include TensorCore utilization, SIMD instruction issue rate, cache hit rate, bandwidth utilization, communication hiding rate, scheduling overhead, and queue congestion, comprehensively reflecting hardware execution efficiency and system scheduling efficiency. When a decrease in hardware utilization is detected, the system adjusts block size, re-aggregation granularity, fragmentation strategy, or issue strategy, rather than dismantling the internal structure of the optimal standard hardware block. This feedback mechanism adheres to the principle of hardware efficiency conservation, meaning that scheduling optimization must not come at the expense of intra-block execution efficiency. By establishing a monitorable, feedback-enabled, and adaptively optimizable execution loop, the system can dynamically adjust its scheduling strategy based on actual operating metrics, maintaining stable hardware utilization under scenarios of distributed scale changes, load fluctuations, or hardware differences, while adhering to the bottom-line constraint of not dismantling the internal structure of the standard hardware block, ensuring long-term operational efficiency stability.

[0062] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A block-granularity graph scheduling method that guarantees hardware efficiency conservation, characterized in that, Includes the following steps: Obtain input computing task and hardware capability information, including hardware model, number of computing units, native matrix block size, memory alignment requirements, and available execution flow information; Based on the hardware capability information, select the optimal hardware standard block size and divide the input matrix into one or more optimal hardware standard blocks that satisfy address alignment constraints and have continuous internal elements. Identify block-level sparse structures, remove all-zero blocks from the graph structure, and preserve non-zero blocks as regular dense blocks; Map the matrix block rows to left nodes, the common dimension blocks to middle nodes, and the matrix block columns to right nodes to construct a block-level bipartite graph. Map the matrix blocks to graph edges, and each second-order path corresponds to a block-level multiply-accumulate computation task. Block-level computing tasks, data loading tasks, communication tasks, DMA copy tasks, prefetch tasks, and write-back tasks are uniformly modeled as DAG nodes, and dependency edges and resource vectors are labeled. Distributed graph cutting is performed based on intermediate nodes or common dimension blocks, distributing block-level path tasks to different devices without tearing any hardware-optimal standard blocks. In each time slice, the largest parallel set without data dependencies and resource conflicts is selected from the set of ready nodes as the resource-constrained maximum antichain; The tasks in the maximum anti-chain are asynchronously launched to independent execution streams according to resource type, and minimal event synchronization is inserted only when there is a strong rigid data dependency. The native hardware computing core is invoked to perform multiplication and addition operations within the block, but graph traversal logic and irregular index jumps are not executed within the block. Accumulate and aggregate local results from different parts of the same output block.

2. The block-granularity graph scheduling method for guaranteeing hardware efficiency conservation according to claim 1, characterized in that, The selection of the optimal standard block size in hardware includes: prioritizing the block size supported by the hardware's native tensor units; ensuring that the block start address satisfies the hardware alignment boundary; if the matrix size cannot be divided by the block size, then padding the boundary blocks or selecting a smaller standard block; if a sparse structure exists, then prioritizing the block size that can retain dense non-zero blocks.

3. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The construction of the block-level bipartite graph includes: mapping the block rows of matrix A to left-hand nodes U, mapping the common dimension blocks to middle nodes K, and mapping the block columns of matrix B to right-hand nodes V; mapping the non-zero blocks of A to edges from U to K, and mapping the non-zero blocks of B to edges from K to V; and generating a block-level computation task for each existing second-order path, the input of which is a complete block and the output is a local result block.

4. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The unified modeling of DAG nodes includes: block data loading nodes, block-level GEMM computing nodes, local accumulation nodes, cross-device communication nodes, DMA copy nodes, result reduction nodes, result write-back nodes, prefetching nodes, and cache residency and release nodes; DAG edges represent strong rigid data dependencies, weak temporal process dependencies, resource mutual exclusion dependencies, data version dependencies, and address lifecycle dependencies.

5. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The distributed graph partitioning is performed while satisfying the following constraints: partitioning is performed only on intermediate nodes or the set of path tasks centered on intermediate nodes; no hardware-optimal standard blocks are partitioned; each device executes the complete block GEMM; cross-device communication occurs only during the block input routing and local result aggregation stages; and deterministic aggregation is performed on multiple partitions of the same output block.

6. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The selection of the maximum anti-chain of resource constraints includes: maintaining a set of ready tasks in the current time slice; for any two tasks, if they do not have direct or indirect dependencies in the DAG, do not conflict in read and write addresses, the sum of resource vectors does not exceed the hardware resource limit, and do not violate block integrity and data version consistency, they can be placed in the same cross-section.

7. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 6, characterized in that, After selecting the maximum antichain, hardware affinity re-aggregation is performed: if the block-level tasks are too fine-grained, multiple block-level tasks with the same device, the same precision, the same block size, adjacent memory addresses and no strong dependency conflicts are re-aggregated into hardware-friendly batch execution units. The batch tasks still maintain independence, integrity, continuity and alignment within each block.

8. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The asynchronous transmission according to resource type includes: transmitting computing tasks to a dedicated computing stream to perform tensor unit block computation; transmitting DMA copy tasks to a DMA copy stream to perform memory transfer; transmitting communication tasks to a network communication stream to perform RDMA or NVLink transmission; transmitting prefetch tasks to a prefetch stream to preload the data required for the next time slice; and transmitting result reduction tasks to a result reduction stream to complete local accumulation and cross-card reduction.

9. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The call to the hardware native computing core to perform multiplication and addition operations within the block satisfies the following: no graph traversal logic is executed within the block; no irregular index jumps are executed within the block; the original data layout is not changed within the block; elements are not rearranged due to graph fragmentation within the block; hardware native matrix multiplication or vectorized instructions are called within the block; and the output local result blocks remain continuous, aligned, and aggregateable.

10. The block-granularity graph scheduling method for ensuring hardware efficiency conservation according to claim 1, characterized in that, The feedback adjustment of the collection and execution indicators includes: collecting Tensor Core utilization, SIMD instruction issuance rate, cache hit rate, bandwidth utilization, communication hiding rate, scheduling overhead, and queue blocking status; if a decrease in hardware utilization is detected, the block size, re-aggregation granularity, fragmentation strategy, or issuance strategy is adjusted, rather than splitting the internal structure of the optimal standard block of the hardware.

Images