Method and system for rear-end dynamic calculation graph division and scheduling of heterogeneous hardware

Abstract

The invention discloses a rear-end dynamic calculation graph dividing and scheduling method and system for heterogeneous hardware. The method comprises the steps that S1, a scheduler is initialized; s2, a multi-round heuristic rear-end distribution stage; s3, performing logic segmentation on the computational graph based on a back-end distribution result, and inserting necessary cross-back-end data copy nodes; s4, the scheduler generates an independent sub-graph representation for each segmentation segment of the segmentation so as to support memory pre-allocation; and S5, scheduling calculation tasks according to a segmentation sequence in a final execution stage. By adopting the method, the sub-graphs can be automatically divided and the rear ends can be automatically allocated according to the capability of the hardware rear end and the model graph structure during operation, the cross-rear-end tensor copy can be created when necessary, and the memory and the execution process can be organized and optimized, so that higher performance and resource utilization rate can be obtained in a heterogeneous hardware environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer systems and parallel computing technology, in particular to a method and system for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends. BACKGROUND

[0002] With the growing size of deep learning models, the choice of parallel inference / training hardware is also increasingly diverse. Different hardware backends differ significantly in computing power, supported operator types, memory layout and buffer types, bandwidth, host / device memory isolation properties, etc. Existing methods usually have the following problems or limitations:

[0003] Single-backend execution or static partitioning: the model is usually statically placed on a single device or manually partitioned by engineers in advance, lacking runtime adaptability and failing to dynamically utilize the advantages of multiple backends;

[0004] Manual partitioning is complex: manual partitioning consumes a lot of engineering time and is difficult to adjust to input data changes, device resource changes, and runtime conditions;

[0005] Buffer / layout inconsistency: differences in buffer type and tensor layout support between different backends result in a large amount of explicit copying at runtime or degradation to lower performance backends;

[0006] Operation support differences: some backends do not support all operators, or support a subset or more efficient implementation, requiring backend support detection and fallback strategies. SUMMARY

[0007] The technical task of the present application is to address the above deficiencies, and to provide a method and system for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends, which can automatically partition subgraphs, allocate backends, create cross-backend tensor copies when necessary, and organize and optimize memory and execution processes based on the capabilities of hardware backends and model graph structures at runtime, to achieve higher performance and resource utilization in a heterogeneous hardware environment.

[0008] In a first aspect, the present application provides a method for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends, comprising the following steps:

[0009] S1: initializing a scheduler, resetting the internal state of the scheduler, creating a temporary context for temporary tensor management by the scheduler, and associating a memory pool with the temporary context;

[0010] S2: multi-round heuristic backend allocation phase, the scheduler allocates the backend to the nodes of the computation graph according to the pre-allocation information, weight position preference and operation compatibility; propagates the allocated backend label according to the order of the computation graph nodes, preferentially expands the high-priority backend, and skips the CPU to reduce subgraph partitioning; attempts to migrate the nodes allocated to the low-priority backend to the higher-priority backend; synchronously allocates the unallocated nodes and the associated nodes to the same backend through the source / view backtracking mechanism; and finally allocates the backend that can support the operation and maximizes the compatibility with the existing input tensor backend to the nodes that still cannot be allocated.

[0011] S3: logically splits the computation graph based on the backend allocation result, and inserts necessary cross-backend data copy nodes;

[0012] S4: the scheduler generates an independent subgraph representation for each split segment of the split to support memory pre-allocation;

[0013] S5: the final execution phase schedules the computation tasks according to the order of the split segments.

[0014] Optionally, the scheduler includes at least the following key data structures: a backend set and the buffer type, priority information of each backend; a backend id mapping of each tensor; and a partition array for recording the partition result of dividing the dynamic computation graph to each backend.

[0015] One optional approach is that, in S2, the scheduler performs preliminary backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility. This includes: firstly, prioritizing the allocation of operators containing tensors to pre-allocated backends or weight backends; traversing all leaf nodes; for leaf nodes that have not yet been allocated backends, if they already hold buffers, the scheduler selects the highest priority backend that supports the buffer and the corresponding operation; for leaf nodes marked as external inputs, they are assigned to CPUs by default as a fallback strategy; for leaf nodes without buffers but referenced by other node views, they inherit the view backend to maintain consistency; subsequently, the scheduler traverses the remaining non-leaf nodes according to the topological order or execution order of the computation graph; propagating the allocated backend labels according to the order of the computation graph nodes, prioritizing the expansion of high-priority backends, including: during forward traversal, the scheduler maintains a current backend variable, recording the ID of the most recent non-CPU backend; when encountering... If an unassigned node is supported by the current backend, it is assigned to the current backend. Backward traversal captures upstream mergeable nodes from the tail of the computation graph. After the non-CPU backends are expanded, the scheduler performs a round of expansion on the CPU backends to fill remaining nodes. Attempts are made to migrate nodes already assigned to low-priority backends to higher-priority backends, including: the target high-priority backend and the current backend use the same buffer type, and all input tensors of the node are effectively supported under the target high-priority backend. Through a source / view backtracking mechanism, unassigned nodes and associated nodes are synchronously assigned to the same backend, including: for nodes with view sources, if they are not yet assigned, they directly inherit the backend of the view source node; for all input sources of the current node, if an input source is not yet assigned, and if the input source itself is a view, it is assigned to the backend of its view source; otherwise, the backend of the current node is used as the assignment target for that input source.

[0016] One optional approach is that, in step S3, the computation graph is logically segmented based on the backend allocation result, and necessary cross-backend data copy nodes are inserted. This includes: traversing the node sequence of the allocated backend, dividing nodes that continuously use the same backend into a segment, with each segment corresponding to a subgraph interval executed on the same backend. During the traversal, if certain conditions are met, the creation of a new segment is triggered: first, the backend of the current node is inconsistent with the backend of the current segment; second, the number of inputs in the current segment exceeds a preset threshold; and third, to avoid memory not being released in time due to referencing weights located on other backends with incompatible types in the same segment. For each input source, if its backend is different from the backend of the current segment and the buffer type is incompatible, the scheduler will create a replica tensor for it on the target backend. The replica tensor adopts the same layout as the original tensor and is explicitly marked as an input / output type. At the same time, the replica tensor is added to the input list of the current segment so that data copying is triggered during execution.

[0017] One optional approach is that, in step S4, the scheduler generates an independent subgraph representation for each segment of the partition to support memory pre-allocation and execution scheduling. This includes: estimating the upper bound of memory required by the computation graph; expanding the internal graph structure array based on the number of nodes, leaf nodes, the number of inputs to each segment, and the number of replicas; for each segment, extracting the corresponding node interval from the source graph to form a subgraph view; calling the graph optimization interface of the target backend to obtain the kernel fusion or memory layout optimization subgraph; and sequentially inserting dependent nodes into the subgraph view.

[0018] One possible approach is to sequentially insert dependent nodes into the subgraph view, including: first adding input dependent nodes to ensure that the original data is not released before the copy is completed; then inserting actual input copy nodes, which will be allocated memory before execution and populated with data through events or direct copying; and finally inserting optimized subgraph computation nodes.

[0019] One optional approach is that, in S4, the scheduler first compares the backend allocation results of the nodes and leaf nodes of the current subgraph with the record of the last execution. If a backend change or buffer type adjustment is detected, it is determined that memory needs to be reallocated. After successful allocation, the buffer and data pointer of each tensor are updated. For view tensors, they are kept to share the original buffer, and only the offset is adjusted to reflect the new layout.

[0020] One possible approach is that, in S5, for each segment: first, all its input copies are processed. If the input originates from an external source, the copy is executed immediately to prevent the user program from overwriting the original data; otherwise, a synchronous copy is executed. If the input originates from an upstream computing node, the copy is executed only after the upstream backend completes its event. Subsequently, the backend's subgraph execution interface is called to run the subgraph. If a callback function exists, conditional execution or partial synchronization can be implemented accordingly. After execution, the event is recorded for downstream segments to wait for or schedule in parallel.

[0021] On the other hand, the present invention provides a dynamic computation graph partitioning and scheduling system for heterogeneous hardware backend, comprising: an initialization scheduler unit for resetting the internal state of the scheduler, wherein the scheduler creates a temporary context for temporary tensor management, and the temporary context is associated with a memory pool;

[0022] A multi-round heuristic backend allocation unit is used by the scheduler to perform initial backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility; propagate the allocated backend labels in the order of computation graph nodes, prioritizing the expansion of high-priority backends while skipping CPUs to reduce subgraph splitting; attempt to migrate nodes already allocated to low-priority backends to higher-priority backends; synchronize unallocated nodes and associated nodes to the same backend through a source / view backtracking mechanism; and finally allocate nodes that still cannot be allocated to backends that support operations and maximize compatibility with existing input tensor backends.

[0023] The subgraph segmentation unit is used to logically segment the computation graph based on the backend allocation results and insert necessary cross-backend data copy nodes;

[0024] A memory allocation unit is used by the scheduler to generate an independent subgraph representation for each segment of the partition to support memory pre-allocation;

[0025] The execution unit is used to schedule computation tasks in the order of segmentation during the final execution phase.

[0026] The method and system for dynamic computation graph partitioning and scheduling for heterogeneous hardware backends of the present invention have the following advantages: runtime automation, which can automatically allocate and partition backends at runtime, reducing the cost of manual partitioning, and can dynamically adapt to changes in device topology or input; improved performance utilization, which prioritizes the allocation of supported operators to high-priority backends and keeps continuous subgraphs on the same backend, reducing cross-backend data migration and improving the utilization rate of computing devices; memory reuse and on-demand copying, which rationally constructs copy nodes and splits, avoiding holding the weight buffer of low-priority devices on high-priority devices for a long time, reducing peak memory requirements; and scalability, which supports heterogeneous expansion of the number, type, and capabilities of backends, and retains backend-customized optimization interfaces (such as backend graph optimization and asynchronous copy interfaces). Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] The invention will be further described below with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart of a method for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends according to Embodiment 1 of the present invention;

[0030] Figure 2 This is a logical structure block diagram of a heterogeneous hardware backend dynamic computing graph partitioning and scheduling system according to Embodiment 2 of the present invention;

[0031] Figure 3 This is a schematic diagram of the working process of a dynamic computation graph partitioning and scheduling system for heterogeneous hardware backend according to Embodiment 2 of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0033] It should be understood that in the description of the embodiments of the present invention, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. In the embodiments of the present invention, "multiple" refers to two or more.

[0034] In this embodiment of the invention, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, or both A and B existing simultaneously. Furthermore, in this document, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0035] Definitions:

[0036] A computation graph consists of a series of operator nodes, which are linked by zero or more inputs and an optional view source (view_src).

[0037] Backend: Represents the running device and default buffer type. The backend has a set of capabilities, including which operations it supports, which buffer types it supports, and whether it supports asynchronous copying / events.

[0038] split: records the subgraph interval [i_start, i_end), the backend_id, and the list of external inputs for this split.

[0039] Example 1

[0040] like Figure 1 As shown, the method for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends provided in this embodiment aims to efficiently partition the computation graph and perform collaborative execution among multiple heterogeneous hardware backends (such as NPU, GPU, TPU, CPU, various accelerators, etc.) to improve throughput, reduce latency and save memory resources.

[0041] This embodiment provides a method for partitioning and scheduling dynamic computing graphs for heterogeneous hardware backends, including the following steps:

[0042] A scheduler component is introduced, containing the following key data structures: a set of backends and information such as the buffer type and priority of each backend; a backend ID mapping for each tensor; and a partition array used to record the partitioning results of the dynamic computation graph to each backend (each element records the hardware backend ID, start and end node index, input tensor list, etc.).

[0043] S1: Initialize the scheduler, reset the internal state of the scheduler, and the scheduler creates a temporary context for temporary tensor management, which is associated with a memory pool.

[0044] During the scheduler's initialization phase, the system first performs a complete internal state reset to ensure a clean and consistent initial environment each time the graph structure is built or evaluated. This includes, but is not limited to, resetting key control variables such as the number of subgraphs the current graph is divided into, the number of graph input tensors, and whether a reset has been completed. Furthermore, to support zero-copy operations and efficient management of temporary tensors, the scheduler creates a dedicated temporary context associated with an independent memory pool. This design aims to isolate the resources required for temporary computations from the memory layout of the main computation graph, preventing the allocation of temporary tensors from interfering with the memory planning of weights or intermediate results in the main graph, thereby ensuring overall execution efficiency and memory stability.

[0045] S2: In the multi-round heuristic backend allocation phase, the scheduler performs initial backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility; it propagates the allocated backend labels in the order of computation graph nodes, prioritizing the expansion of high-priority backends while skipping CPUs to reduce subgraph splitting; it attempts to migrate nodes already allocated to low-priority backends to higher-priority backends; through the source / view backtracking mechanism, it synchronously allocates unallocated nodes and associated nodes to the same backend; for nodes that still cannot be allocated, it selects a backend that supports operations and maximizes compatibility with existing input tensor backends for final allocation.

[0046] The backend allocation strategy employs a multi-round heuristic approach. First, a default backend is determined based on initial rules. That is, if a tensor has already been pre-allocated (e.g., the weights are already in a backend buffer) or is an input tensor, its corresponding operator is preferentially allocated to that backend. If the tensor contains a source of weights, the operator is preferentially allocated to the backend where the weights are located.

[0047] In the first round of backend allocation, the scheduler performs initial backend binding on nodes in the computation graph based on pre-allocation information, weight position preferences, and operation compatibility. This process first traverses all leaf nodes. For each leaf node that has not yet been assigned a backend, if it already holds a buffer (e.g., model weights have been loaded onto a specific device such as a GPU), the scheduler will select the highest priority backend that supports both the buffer type and the corresponding operation. If the leaf node is marked as an external input, it will be assigned to the CPU by default as a fallback strategy to ensure the general accessibility of input data. If the leaf node itself has no buffer but has view references from other nodes, and the view source has been assigned to a backend, it will directly inherit that source backend to maintain the consistency of the data view. Subsequently, the scheduler traverses the remaining non-leaf nodes according to the topological order or execution order of the graph and applies similar rules. If the buffer of a pre-allocated tensor cannot be compatible with any backend that supports the operation, it is considered a serious configuration conflict, and the scheduler will abort the process or trigger a fallback mechanism.

[0048] Then, the expansion propagation proceeds, spreading the assigned backend labels upwards or downwards according to the node order of the computation graph. Prioritizes expanding high-priority backends (such as GPUs) while skipping CPUs to avoid prematurely using them as fallback nodes, thereby reducing subgraph splitting and improving execution coherence.

[0049] The second round of allocation focuses on expanding existing allocations, aiming to generate larger-scale, continuous execution fragments with the same backend, thereby significantly reducing the number of graph partitions and frequent cross-device data migrations. This phase employs a multi-round traversal strategy combining "downward expansion" and "upward expansion": In the forward traversal, the scheduler maintains a current backend variable, recording the ID of the most recent non-CPU backend; when an unallocated node is encountered and the current backend supports its operation, the node is allocated to the current backend; the backward traversal starts from the tail of the graph and executes similar logic to capture upstream mergeable nodes. This process actively skips view-type operations (such as view, reshape, permute, transpose, etc.) because these operations do not generate new data, only change tensor metadata, and their execution position should strictly remain consistent with their source tensor. After completing the expansion of non-CPU backends, the scheduler will also perform a similar expansion traversal on CPU backends to fill in the remaining nodes that can be efficiently processed by the CPU, further optimizing the overall allocation continuity.

[0050] Next, perform an upgrade operation to attempt to migrate nodes that have been assigned to a low-priority backend to a higher-priority backend, provided that the target backend supports the operation, the buffer types are compatible, and all input tensors are available on the target backend.

[0051] The third round of allocation introduces a support-based intelligent decision-making mechanism to handle unassigned nodes and potentially upgrade assigned nodes. For unassigned nodes, the scheduler iterates through all available backends, counting the number of input tensors each backend can directly support (while simultaneously satisfying operation compatibility and buffer type matching), and finally selects the backend with the most support for binding. For assigned nodes, it attempts to upgrade to a higher-priority backend, provided that the target backend uses the same buffer type as the current backend and all input tensors of the node can be effectively supported by the target backend. If these conditions are met, the scheduler updates the backend allocation for the node, thereby improving computational performance without disrupting data flow.

[0052] For nodes that have not yet been assigned, if they have a view relationship (such as operators of the view and permute classes) and the associated nodes have been assigned, they will be synchronously assigned to the same backend through the source / view backtracking mechanism.

[0053] The fourth round of allocation focuses on handling view dependencies, ensuring that all nodes in the graph ultimately have a clearly defined backend. First, for nodes with a `view_src` (view source), if it hasn't been allocated yet, it directly inherits the backend of the view source node to guarantee the physical consistency of view semantics. Second, for all input sources of the current node, if a source hasn't been allocated: if the source itself is a view, it is recursively allocated to the backend of its `view_src`; otherwise, the backend of the current node is used as the allocation target for that source. After these steps, if a very small number of nodes are still unallocated (usually due to extreme compatibility issues), the scheduler will execute a final fallback mechanism: linearly traversing all backends and selecting the first backend that supports the node's operations for binding, thus ensuring the completeness and robustness of the scheduling process.

[0054] Finally, for nodes that still cannot be assigned, select a backend that supports the operation and maximizes compatibility with existing input tensor backends for assignment.

[0055] S3: Logically segment the computation graph based on the backend allocation results and insert necessary cross-backend data copy nodes.

[0056] The subgraph partitioning strategy traverses the node sequence of the assigned backend, dividing nodes that consecutively use the same backend into a split. Each split corresponds to a subgraph interval executed on the same backend. During traversal, a new split is triggered if certain conditions are met: 1) the backend of the current node is inconsistent with the backend of the current split; 2) the number of inputs to the current split exceeds a preset threshold; 3) to avoid memory delays due to referencing weights located on other backends with incompatible types in the same split (e.g., the target backend does not support the data format of the weight), proactive splitting is performed to facilitate memory reclamation. Furthermore, each split explicitly maintains its required cross-backend inputs, i.e., tensors that were originally located on other backends and cannot be directly used by the backend to which the current split belongs, and records copy information for these inputs to complete necessary data migration during execution.

[0057] The core task of this stage is to logically split the computation graph based on the backend allocation results and insert necessary cross-backend data copy nodes. The scheduler uses the backend of the first non-view node as the reference backend for the current split segment and traverses the node sequence sequentially. When any of the following situations are detected, the creation of a new split segment is triggered: (1) the backend of the current node is inconsistent with the backend of the current split segment; (2) the number of inputs already contained in the current split segment exceeds the preset threshold, and the new input is located in another backend and the target backend does not support its buffer type; (3) there are weights distributed in different backends and the target backend cannot natively support the weight format. In order to avoid long-term residence and memory pressure, it is necessary to release it in time through splitting. For each input source, if its backend is different from the backend of the current split segment and the buffer type is incompatible, the scheduler will create a replica tensor for it in the target backend. The replica tensor adopts the same layout as the original tensor and is explicitly marked as input / output type to prevent it from being reused and overwritten in the subsequent memory allocation stage; at the same time, the replica is added to the input list of the current split segment so that data copying is triggered during execution. To avoid redundancy, the system uses a hashing mechanism to ensure that a copy of the same value is created only once for the same backend.

[0058] During the split formation phase, if a node's input tensor comes from a different backend than the target backend, and the target backend does not support the input's buffer type, the system will create a copy of the target backend tensor with a consistent layout or compatible buffer type for that input. To improve execution efficiency, the system prioritizes using the asynchronous replication interface provided by the backend in conjunction with an event mechanism for cross-backend data transfer; if the backend does not support asynchronous replication, it falls back to synchronous replication supplemented by explicit synchronization operations to ensure data consistency.

[0059] S4: The scheduler generates an independent subgraph representation for each segment of the partition to support subsequent memory pre-allocation and execution scheduling.

[0060] When constructing an allocatable subgraph copy, the system preserves necessary dependent nodes (such as view or copy nodes) to ensure the correct lifecycle of the copied tensor and the source tensor, and completes the relevant memory allocation before split execution.

[0061] The scheduler queries the backend allocator for the type and size of the required buffers and allocates memory for tensors in the subgraph copy at the corresponding backend. If a change in the backend or buffer type is detected, the scheduler triggers a reallocation process, ensuring synchronization across backends during this process to prevent conflicts caused by parallel execution. During the execution phase, each split completes the necessary input copying at its respective backend before performing subgraph computation. The execution order across splits is strictly guaranteed through events or synchronization interfaces, while also supporting fine-grained callback mechanisms, allowing further control and optimization of the execution process within the subgraph or at the subrange level.

[0062] After completing the logical partitioning, the scheduler enters the subgraph replica construction phase, generating an independent subgraph representation for each partition to support subsequent memory pre-allocation and execution scheduling. First, the system estimates the upper bound of memory required for the entire graph, taking into account factors such as the number of nodes, leaf nodes, the number of inputs and replicas for each partition, and expands the internal graph structure array accordingly. Then, for each partition: (1) extract the corresponding node intervals in the source graph to form a subgraph view; (2) call the graph optimization interface of the target backend to obtain possible kernel fusion or memory layout optimization versions; (3) insert dependent nodes in the graph in sequence: first add input dependent nodes (as a view of the original input tensor) to ensure that the original data will not be released before the copy is completed; then insert the actual input copy nodes, which will be allocated memory before execution and filled with data through events or direct copying; finally insert the optimized subgraph computation nodes. All leaf nodes of the graph will also be appended to the leaf nodes of the original graph and their backend allocation information will be synchronized.

[0063] The memory allocation phase is responsible for allocating actual device memory for all tensors in the graph. The scheduler first compares the current graph's node and leaf node backend allocation results with the records from the previous execution. If a backend change or buffer type adjustment is detected, it determines that memory needs to be reallocated. After successful allocation, the buffer and data pointers of each tensor are updated; for view tensors, they maintain their shared original buffers, and only the offsets are adjusted to reflect the new layout.

[0064] S5: In the final execution phase, computation tasks are scheduled in the order of segmentation.

[0065] The final execution phase schedules computational tasks according to the segment order. For each segment: first, all its input copies are processed—if the input originates from an external source, to prevent user programs from overwriting the original data, the copy is executed immediately; backends supporting asynchronous copying will use asynchronous interfaces and record completion events. Otherwise, synchronous copying is performed. If the input originates from an upstream computation node, the copying must wait for the upstream backend's completion event before execution. Subsequently, the backend's graph execution interface is called to run the subgraph; if callback functions exist, conditional execution or partial synchronization can be implemented accordingly. After execution, an event is recorded for downstream segments to wait for or schedule in parallel.

[0066] In this embodiment, the original graph is divided into several subgraphs that are executable on the same backend by performing a multi-round heuristic and scalable backend allocation process on the computation graph. Tensor copies that are available to the backend are created when needed to ensure the correct execution and memory lifecycle of the subgraphs.

[0067] Example 2

[0068] like Figure 2 and Figure 3 As shown, this embodiment provides a dynamic computation graph partitioning and scheduling system for heterogeneous hardware backends, which corresponds one-to-one with the method in Embodiment 1, and includes:

[0069] The scheduler unit is initialized to reset the internal state of the scheduler, which creates a temporary context for temporary tensor management, and the temporary context is associated with a memory pool.

[0070] A multi-round heuristic backend allocation unit is used by the scheduler to perform initial backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility; propagate the allocated backend labels in the order of computation graph nodes, prioritizing the expansion of high-priority backends while skipping CPUs to reduce subgraph splitting; attempt to migrate nodes already allocated to low-priority backends to higher-priority backends; synchronize unallocated nodes and associated nodes to the same backend through a source / view backtracking mechanism; and finally allocate nodes that still cannot be allocated to backends that support operations and maximize compatibility with existing input tensor backends.

[0071] The subgraph segmentation unit is used to logically segment the computation graph based on the backend allocation results and insert necessary cross-backend data copy nodes;

[0072] A memory allocation unit is used by the scheduler to generate an independent subgraph representation for each segment of the partition to support memory pre-allocation;

[0073] The execution unit is used to schedule computation tasks in the order of segmentation during the final execution phase.

[0074] The method and system for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the method and system for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the content of the appended claims.

Claims

1. A method for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends, characterized in that, Includes the following steps: S1: Initialize the scheduler, reset the internal state of the scheduler, and the scheduler creates a temporary context for temporary tensor management, which is associated with a memory pool; S2: In the multi-round heuristic backend allocation phase, the scheduler performs initial backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility; it propagates the allocated backend labels in the order of computation graph nodes, prioritizing the expansion of high-priority backends while skipping CPUs to reduce subgraph splitting; it attempts to migrate nodes already allocated to low-priority backends to higher-priority backends; through a source / view backtracking mechanism, it synchronously allocates unallocated nodes and associated nodes to the same backend; for nodes that still cannot be allocated, it selects a backend that supports operations and maximizes compatibility with existing input tensor backends for final allocation; S3: Logically segment the computation graph based on the backend allocation results and insert necessary cross-backend data copy nodes; S4: The scheduler generates an independent subgraph representation for each segment of the partition to support memory pre-allocation; S5: In the final execution phase, computation tasks are scheduled in the order of segmentation.

2. The method for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends according to claim 1, characterized in that, The scheduler includes at least the following key data structures: the backend set and the buffer type and priority information of each backend; the backend ID mapping for each tensor; and a partition array used to record the partitioning results of the dynamic computation graph to each backend.

3. The method for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends according to claim 1, characterized in that, In S2, the scheduler performs preliminary backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility. This includes: firstly, prioritizing the allocation of operators containing tensors to pre-allocated backends or backends containing weights; traversing all leaf nodes; for leaf nodes without assigned backends, if they already hold buffers, the scheduler selects the highest priority backend that supports the buffer and the corresponding operation; for leaf nodes marked as external inputs, they are assigned to CPUs by default as a fallback strategy; for leaf nodes without buffers but referenced by other node views, the view backend is inherited to maintain consistency; subsequently, the scheduler traverses the remaining non-leaf nodes according to the topological or execution order of the computation graph; propagating the assigned backend labels according to the order of the computation graph nodes, prioritizing the expansion of high-priority backends, including: during forward traversal, the scheduler maintains a current backend variable, recording the ID of the most recent non-CPU backend; when encountering an unassigned node... When the current backend supports its operation, the node is assigned to the current backend; backward traversal captures upstream mergeable nodes from the tail of the computation graph; after the non-CPU backend is expanded, the scheduler performs a round of expansion on the CPU backend to fill the remaining nodes; attempts are made to migrate nodes already assigned to low-priority backends to higher-priority backends, including: the target high-priority backend and the current backend use the same buffer type, and all input tensors of the node can be effectively supported under the target high-priority backend; through the source / view backtracking mechanism, unassigned nodes and associated nodes are synchronously assigned to the same backend, including: for nodes with view sources, if they have not yet been assigned, they directly inherit the backend of the view source node; for all input sources of the current node, if an input source has not yet been assigned, and if the input source itself is a view, it is assigned to the backend of its view source; otherwise, the backend of the current node is used as the assignment target for the input source.

4. The method for dynamic computation graph partitioning and scheduling of heterogeneous hardware backends according to claim 1, characterized in that, In step S3, the computation graph is logically segmented based on the backend allocation results, and necessary cross-backend data copy nodes are inserted. This includes: traversing the node sequence of the allocated backends, dividing nodes that continuously use the same backend into a segment, with each segment corresponding to a subgraph interval executed on the same backend. During the traversal, if certain conditions are met, the creation of a new segment is triggered: first, the backend of the current node is inconsistent with the backend of the current segment; second, the number of inputs in the current segment exceeds a preset threshold; and third, to avoid memory not being released in time due to referencing weights located on other backends with incompatible types in the same segment. For each input source, if its backend is different from the backend of the current segment and the buffer type is incompatible, the scheduler will create a replica tensor for it on the target backend. The replica tensor adopts the same layout as the original tensor and is explicitly marked as an input / output type. At the same time, the replica tensor is added to the input list of the current segment so that data copying is triggered during execution.

5. The method for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends according to claim 4, characterized in that, In step S4, the scheduler generates an independent subgraph representation for each segment of the partition to support memory pre-allocation and execution scheduling. This includes: estimating the upper bound of memory required by the computation graph; expanding the internal graph structure array based on the number of nodes, leaf nodes, the number of inputs to each segment, and the number of replicas; for each segment, extracting the corresponding node interval from the source graph to form a subgraph view; calling the graph optimization interface of the target backend to obtain the kernel fusion or memory layout optimization subgraph; and sequentially inserting dependent nodes into the subgraph view.

6. The method for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends according to claim 5, characterized in that, In the subgraph view, dependent nodes are inserted sequentially, including: first, adding input dependent nodes to ensure that the original data is not released before the copy is completed; then, inserting the actual input copy nodes, which will be allocated memory before execution and populated with data through events or direct copying; and finally, inserting the optimized subgraph computation nodes.

7. The method for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends according to claim 5, characterized in that, In S4, the scheduler first compares the backend allocation results of the nodes and leaf nodes of the current subgraph with the record of the last execution. If a backend change or buffer type adjustment is detected, it is determined that memory needs to be reallocated. After successful allocation, the buffer and data pointer of each tensor are updated. For view tensors, they share the original buffer and only the offset is adjusted to reflect the new layout.

8. The method for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends according to claim 1, characterized in that, In S5, for each segment: first, all its input copies are processed. If the input comes from an external source, the copy is executed immediately to prevent the user program from overwriting the original data; otherwise, a synchronous copy is executed. If the input comes from an upstream computing node, the copy is executed only after the upstream backend completes its event. Subsequently, the subgraph execution interface of the backend is called to run the subgraph; if there is a callback function, conditional execution or partial synchronization can be implemented accordingly. After execution, the event is recorded for downstream segmentation waiting or parallel scheduling.

9. A system for dynamic computation graph partitioning and scheduling in heterogeneous hardware backends, characterized in that, include: The scheduler unit is initialized to reset the internal state of the scheduler, which creates a temporary context for temporary tensor management, and the temporary context is associated with a memory pool. A multi-round heuristic backend allocation unit is used by the scheduler to perform initial backend binding on computation graph nodes based on pre-allocation information, weight position preferences, and operation compatibility; propagate the allocated backend labels in the order of computation graph nodes, prioritizing the expansion of high-priority backends while skipping CPUs to reduce subgraph splitting; attempt to migrate nodes already allocated to low-priority backends to higher-priority backends; synchronize unallocated nodes and associated nodes to the same backend through a source / view backtracking mechanism; and finally allocate nodes that still cannot be allocated to backends that support operations and maximize compatibility with existing input tensor backends. The subgraph segmentation unit is used to logically segment the computation graph based on the backend allocation results and insert necessary cross-backend data copy nodes; A memory allocation unit is used by the scheduler to generate an independent subgraph representation for each segment of the partition to support memory pre-allocation; The execution unit is used to schedule computation tasks in the order of segmentation during the final execution phase.

Images