Graph-Optimized Compilation Method and Apparatus for Multi-GPU Heterogeneous Inference

Abstract

This invention provides a graph-optimized compilation method and apparatus for multi-GPU heterogeneous inference, relating to the field of computer technology. The method includes: acquiring the hardware parameters, interconnect topology information, and computation task graph of the target multi-GPU system; performing data flow analysis on the computation task graph to determine the operator types in the graph; determining a parallel strategy based on the operator types; partitioning the computation task graph according to the hardware parameters and interconnect topology information to determine multiple sub-tasks and their corresponding target computing cards; in a unified intermediate representation layer of the target multi-GPU system, performing tensor partitioning on each of the multiple sub-tasks according to the interconnect topology information and parallel strategy, and inserting cross-GPU communication nodes between the sub-tasks to generate an executable file that is uniformly loaded and executed on the target multi-GPU system, enabling the target computing card to load and execute the corresponding sub-tasks. This invention improves multi-GPU inference efficiency through an automatic parallel strategy for heterogeneous topology and unified compilation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and for example to a graph-optimized compilation method and apparatus for multi-GPU heterogeneous inference. Background Technology

[0002] With the rapid development of fields such as artificial intelligence, scientific computing, and big data processing, the computing power of a single card is no longer sufficient to meet the ever-increasing computing demands. Combining multiple identical computing cards, such as Graphics Processing Units (GPUs) and Neural Processing Units (NPUs), into a multi-card computing system using high-speed interconnect technologies such as NVLink, PCIe (Peripheral Component Interconnect Express), InfiniBand, and Ethernet has become an important means of improving computing power.

[0003] In related technologies, most existing software compilation tools are designed for single cards or single nodes, failing to fully leverage the overall performance of multi-card interconnect systems. Current multi-card system development typically employs manual programming, requiring developers to explicitly use communication libraries such as NCCL (NVIDIA Collective Communications Library) and MPI (Message Passing Interface) for inter-card data exchange and synchronization, and manually allocate computational tasks across the cards. This approach demands a deep understanding of the underlying interconnect topology and communication overhead, resulting in high programming complexity, a high risk of errors, and difficulty in achieving optimal task partitioning and communication hiding. Consequently, the current method suffers from low compilation efficiency. Summary of the Invention

[0004] The present invention aims to provide a graph-optimized compilation method and apparatus for multi-GPU heterogeneous inference.

[0005] According to one aspect of the present invention, a graph optimization compilation method for multi-GPU heterogeneous inference is proposed, comprising: acquiring hardware parameters, interconnect topology information, and computation task graph of the target multi-GPU system, and performing data flow analysis on the computation task graph to determine the operator types in the computation task graph; determining a parallel strategy based on the operator types; performing task partitioning on the computation task graph according to the hardware parameters and interconnect topology information to determine multiple sub-tasks and their corresponding target computing cards; performing tensor partitioning on each of the multiple sub-tasks in the unified intermediate representation layer of the target multi-GPU system according to the interconnect topology information and parallel strategy, and inserting cross-GPU communication nodes between the multiple sub-tasks to generate an executable file that is uniformly loaded and executed on the target multi-GPU system, so that the target computing card loads and executes the corresponding sub-task.

[0006] According to one aspect of the present invention, a graph-optimized compiler apparatus for multi-GPU heterogeneous inference is proposed, comprising:

[0007] The operator type determination module is used to obtain the hardware parameters, interconnect topology information and computing task graph of the target multi-card system, and to perform data flow analysis on the computing task graph to determine the operator type in the computing task graph; The parallel strategy determination module is used to determine the parallel strategy based on the operator type. The task partitioning module is used to partition the computing task graph according to hardware parameters and interconnect topology information to determine multiple subtasks and their respective target computing cards. The tensor partitioning module is used to partition multiple subtasks into tensors based on interconnect topology information and parallel strategies in the unified intermediate representation layer of the target multi-card system. It also inserts cross-card communication nodes between the subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card can load and execute the corresponding subtask.

[0008] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to perform the method described above.

[0009] According to one aspect of the invention, a non-transitory computer-readable medium is provided, on which readable instructions are stored, which, when executed by a processor, cause the processor to perform the method described above.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention.

[0011] Beneficial effects: Through the embodiments provided by this invention, by acquiring hardware parameters and interconnection topology information and combining data flow analysis to automatically determine parallel strategies, the manual partitioning process that relies on human experience in traditional development is transformed into automatic mapping based on cost models. This allows operator partitioning to accurately match the computing power differences and link bandwidth bottlenecks of each computing card, thereby avoiding performance degradation caused by uneven load or communication blockage from the source. Tensor partitioning and insertion of cross-card communication nodes in the unified intermediate representation layer are key to achieving compilation and runtime collaboration. Instead of generating isolated machine code for a single card, a unified executable file containing complete communication semantics is generated. This eliminates the need for explicit calls to external communication libraries at runtime, as the compiler pre-plans the data flow path, achieving deep integration and overlap of communication and computation, and significantly reducing inference latency. Since the above optimizations are all statically completed during the compilation phase, the generated executable file has been optimized for specific heterogeneous topologies. This not only eliminates the need for developers to manually write complex synchronization logic, lowering the development threshold, but also improves the overall compilation efficiency of the unified compilation system through global task scheduling and resource allocation. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without exceeding the scope of protection claimed by the present invention.

[0013] Figure 1 This is a schematic diagram of the architecture of a unified compilation system for multi-card heterogeneous inference provided in an embodiment of the present invention; Figure 2 A schematic diagram of memory management provided for an embodiment of the present invention; Figure 3 This is a schematic diagram of the process for generating executable instructions provided in an embodiment of the present invention; Figure 4 A flowchart illustrating a unified compilation method for multi-GPU heterogeneous inference provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of computing card information provided in an embodiment of the present invention; Figure 6 A block diagram of a unified compilation device for multi-card heterogeneous inference provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0015] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0016] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0017] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0018] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0019] The English abbreviations used in this invention and their corresponding Chinese translations are as follows: The English abbreviation for GPU is Graph Processing Unit. The English abbreviation is NPU, which translates to Neural Processing Unit. English abbreviation: PCIe, Chinese translation: Peripheral Component Interconnect Express, English full name: Peripheral Component Interconnect Express; English abbreviation: NCCL, Chinese translation: NVIDIA Collective Communications Library, English full name: NVIDIA Collective Communications Library; English abbreviation: MPI, Chinese translation: Message Passing Interface, English full name: Message Passing Interface; English abbreviation: IR, Chinese translation: Intermediate Representation, English full name: Intermediate Representation; English abbreviation: AST, Chinese translation: Abstract Syntax Tree, English full name: Abstract Syntax Tree; English abbreviation: ONNX, Chinese translation: Open Neural Network Exchange, English full name: Open Neural Network Exchange; English abbreviation: MLIR, Chinese translation: Multi-Level Intermediate Representation, English full name: Multi-Level Intermediate Representation; English abbreviation: DP, Chinese translation: Data Parallelism, English full name: Data Parallelism; English abbreviation: TP, Chinese translation: Tensor Parallelism, English full name: Tensor Parallelism; English abbreviation: PP, Chinese translation: Pipeline Parallelism, English full name: Pipeline Parallelism; English abbreviation: API, Chinese translation: Application Programming Interface, English full name: Application Programming Interface; English abbreviation: PCI, Chinese translation: Peripheral Component Interconnect, English full name: Peripheral Component Interconnect; English abbreviation: NVLink, Chinese translation: NVIDIA Link, English full name: NVIDIA Link; English abbreviation: PCIe Switch, Chinese translation: Peripheral Component Interconnect Express Switch, English full name: Peripheral Component Interconnect Express Switch; English abbreviation: CUDA, Chinese translation: Unified Compute Device Architecture, English full name: Compute Unified Device Architecture; English abbreviation: CPU, Chinese translation: Central Processing Unit, English full name: Central Processing Unit; English abbreviation: DSP, Chinese translation: Digital Signal Processor, English full name: Digital Signal Processor; English abbreviation: ASIC, Chinese translation: Application Specific Integrated Circuit, English full name: Application Specific Integrated Circuit; English abbreviation: FPGA, Chinese translation: Field Programmable Gate Array, English full name: Field Programmable Gate Array; English abbreviation: EISA, Chinese translation: Extended Industry Standard Architecture, English full name: Extended Industry Standard Architecture; English abbreviation: ROM, Chinese translation: Read Only Memory, English full name: Read Only Memory English abbreviation: RAM, Chinese translation: Random Access Memory, English full name: Random Access Memory English abbreviation: EEPROM, Chinese translation: Electrically Erasable Programmable Read Only Memory, English full name: Electrically Erasable Programmable Read Only Memory English abbreviation: CD ROM, Chinese translation: Compact Disc Read Only Memory, English full name: Compact Disc Read Only Memory.

[0020] Figure 1This is a schematic diagram of the architecture of a unified compilation system for multi-GPU heterogeneous inference provided in an embodiment of the present invention. The compilation system of the present invention adopts a five-layer architecture: a unified front-end module, a high-level IR (Intermediate Representation) generation and lowering module, a low-level IR generation and lowering module, a hardware back-end module, and a unified runtime module. Users input model files or operator code into the unified front-end module for syntax parsing, type inference, semantic checking, and generation of a unified AST (Abstract Syntax Tree). The information output from the unified front-end module is sent to the high-level IR generation and lowering module for quantization, graph optimization, and graph sorting. The processed information is then sent to the low-level IR generation and lowering module for optimization using a global joint optimizer and multi-GPU collaborative optimization. The optimized information is then sent to the hardware back-end module for hardware-related optimization and dynamic tensor shape inference. Finally, the processed information is sent to the unified runtime module for task scheduling and unified memory management.

[0021] The unified front-end module can accept inputs such as TorchScript, ONNX (Open Neural Network Exchange) format model files, and Triton operator code. This module performs sign checking, type inference, and unified AST processing, outputting a hardware- and framework-independent structured computational IR representation. The high-level IR generation and lowering module can insert quantization operators to accelerate computation. In graph optimization, it can perform dead code elimination, common subexpression elimination, operator fusion, and constant folding operations. In graph sorting, it can sort by breadth-first or depth-first search, while grouping operators that map to the same computational units together as much as possible. The high-level IR generation and lowering module can handle extended MLIR (Multi-Level Intermediate Representation), adding instructions for full collection, full reduction, and data copying, carrying topology, memory offset, sharding, communication, and dependency information; it also includes a global joint optimizer.

[0022] The global joint optimizer can perform global data layout optimization. Specifically, it can optimize the operator data layout in the model according to hardware computing requirements. 1. Divide operators into two categories: the first category is one-to-one input and output, and the second category is one-to-many, many-to-one, or many-to-many.

[0023] 2. The layout format of the second type of operator can be combined with the layout format of the first type.

[0024] Specifically, it can identify one-to-many, many-to-one, and many-to-many operators in the user input information, and generate a group for each such operator. The first operator in the group is a one-to-many, many-to-one, or many-to-many operator. A greedy method is used to traverse downwards; if a one-to-one operator is encountered, it is inserted into the group; otherwise, the traversal stops. Memory placement operators between computational operators within the group are eliminated.

[0025] A diagram illustrating memory management can be found here. Figure 2 Memory / cache allocation includes communication-shared caches, which resolve shared memory consistency issues through hardware and software coordination.

[0026] For multi-card collaborative optimization, cross-card tasks can be divided. First, the number of cards and the topology are obtained. Automatic selection is then performed based on layer type: attention mechanism - DP (Data Parallelism), linear transformation / convolution operation - TP (Tensor Parallelism), large model deep layers - PP (Pipeline Parallelism), large images - TP. At the IR layer, the computation graph is segmented: segmentation operator, segmentation weights, segmentation feature map, and insertion of communication nodes (full collection / reduction divergence / full reduction). In the case of full collection, it is necessary to ensure that the data transmitted from other cards corresponds one-to-one with the weights. Synchronization elimination: identify independent cross-card execution and remove redundant barriers.

[0027] The hardware backend module can achieve hardware-related optimizations through vectorization, loop unrolling / blocking, and instruction scheduling. It supports runtime API (Application Programming Interface) calls by dynamically typed operators to generate executable instructions online, enabling dynamic tensor shape inference. (See reference...) Figure 3 The flowchart shown illustrates the process of generating executable instructions. It calls the dynamic operator processing interface to perform dynamic operator processing, executes tasks in blocks, and generates executable instructions.

[0028] For a unified runtime module, a unified virtual address space can be used, with automatic migration and copying; memory pools for input, output, weights, and temporary areas are allocated according to compiler requirements. Execution is asynchronous and event-driven, following the compiler dependency graph.

[0029] For specific implementation details, please refer to the following examples.

[0030] Figure 4 This is a flowchart illustrating a unified compilation method for multi-GPU heterogeneous inference provided in an embodiment of the present invention. Figure 4 As shown, the method includes steps S10, S11, S12 and S13.

[0031] In step S40, the hardware parameters, interconnection topology information and computing task graph of the target multi-card system are obtained, and data flow analysis is performed on the computing task graph to determine the operator type in the computing task graph.

[0032] In this invention, a computational task graph input by the user can be obtained. This computational task graph can be a model file or operator code. The hardware parameters of the target multi-GPU system can include the number and model of the computing cards, and the interconnection topology information can include the connection method and communication bandwidth between the computing cards. The compiler traverses the computational graph, performs semantic analysis on each node, and the operator types can include attention operators, linear operators, convolution operators, and large image processing operators, etc.

[0033] In some implementations, four AI (Artificial Intelligence) cards of the same model can be used as computing cards, such as a pre-developed CAISA accelerator card, which supports PCI (Peripheral Component Interconnect) switch interconnection. The connection relationships can be found in [reference needed]. Figure 5 .

[0034] In step S41, a parallel strategy is determined based on the operator type.

[0035] In this invention, parallel strategies corresponding to different operator types can be preset, and these parallel strategies can include DP, PP, and TP.

[0036] In some implementations, a parallel strategy can be formulated based on the operator type identified in step S40 and the hardware topology information.

[0037] Attention / LayerNorm type operators: Due to their computational characteristics involving global interactions and relatively controllable memory usage, the strategy tends to be data parallel (DP). That is, the model is copied on multiple cards to process different batches of data, and communication only occurs in gradient synchronization or specific reduction steps.

[0038] Linear / Conv type operators: These operators have huge weight matrices and their computation is highly block-based. The strategy tends towards tensor parallelism (TP). The compiler decides to split the weight matrix by rows or columns onto different computation cards, allowing multiple cards to collaborate on a single matrix operation.

[0039] Large Models / Deep Networks / Large Images: When the model has extremely deep layers or the image resolution is extremely high, causing a single GPU memory to be unable to accommodate the entire computation flow, the strategy tends to be pipelined parallelism (PP). The compiler divides the model into layers, with GPU 0 processing the first few layers and GPU 1 processing the last few layers, forming a pipelined operation.

[0040] For complex modern neural networks, a single strategy is not suitable; a hybrid strategy can be generated. For example, a TP (Progressive Strategies) strategy can be used in the first half of the model to accelerate computation, while a DP (Dynamic Strategies) strategy can be superimposed when the batch processing dimension is large. During the process, the theoretical execution time under different strategies can be estimated, and the strategy combination that minimizes the total time can be selected.

[0041] In step S42, the computing task graph is divided into tasks based on hardware parameters and interconnect topology information to determine multiple sub-tasks and their respective target computing cards.

[0042] This invention allows for pre-setting a task partitioning method, which is associated with hardware parameters and interconnect topology information. It also allows for pre-setting a subtask allocation method. Based on the hardware parameters and interconnect topology information, the corresponding pre-set task partitioning method is located, and the computation task graph is partitioned to obtain multiple subtasks. These subtasks can then be allocated to their respective target computing cards according to the pre-set allocation method.

[0043] In step S43, in the unified intermediate representation layer of the target multi-card system, tensor partitioning is performed on each of the multiple subtasks according to the interconnection topology information and parallel strategy, and cross-card communication nodes are inserted between the multiple subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask.

[0044] This invention enables tensor partitioning of subtasks on a unified intermediate representation layer. Different parallel strategies correspond to different partitioning methods, which may include weight partitioning and feature map partitioning. Because the computation graph is physically partitioned, logical breakpoints appear in the data flow. These breakpoints need to be repaired by explicitly inserting communication operators into the unified IR, which may include: All-Gather Insertion: When a subtask split by TP requires complete input data for computation, or when the results need to be summarized after computation, insert the All-Gather instruction in IR.

[0045] Reduce-Scatter / All-Reduce Insertion: In backpropagation or scenarios requiring global reduction (such as gradient synchronization under DP strategy), insert All-Reduce instructions to sum and broadcast the computation results of multiple cards; or insert Reduce-Scatter in the TP output stage to shard and summarize the results.

[0046] Dependency and Synchronization Management: Analyze the data dependency graph and insert necessary synchronization barriers or event instructions before and after communication nodes to ensure that the data sender sends data only after the data is ready and the receiver starts calculation only after the data arrives. Redundant synchronization instructions are automatically deleted to reduce waiting time.

[0047] The unified IR, after the above processing, is converted into a binary file containing instructions from all compute cards by the backend code generator. This file is loaded at runtime, and each compute card automatically collaborates to complete complex inference or training tasks based on a pre-defined dependency graph and communication instructions.

[0048] This invention automatically determines parallel strategies by acquiring hardware parameters and interconnect topology information, combined with data flow analysis. It transforms the manual partitioning process, which relies on experience in traditional development, into an automatic mapping based on a cost model. This allows operator partitioning to accurately match the computing power differences and link bandwidth bottlenecks of each computing card, thus preventing performance degradation caused by uneven load or communication congestion at the source. Tensor partitioning and cross-card communication node insertion at the unified intermediate representation layer are key to achieving compilation and runtime collaboration. Instead of generating isolated machine code for a single card, it generates a unified executable file containing complete communication semantics. This eliminates the need for explicit calls to external communication libraries at runtime; instead, the compiler pre-plans the data flow path, achieving deep integration and overlap of communication and computation, significantly reducing inference latency. Since all the above optimizations are statically completed during the compilation phase, the generated executable file is optimized for specific heterogeneous topologies. This not only eliminates the need for developers to manually write complex synchronization logic, lowering the development threshold, but also improves the overall compilation efficiency of the unified compilation system through global task scheduling and resource allocation.

[0049] According to some embodiments, when the operator type is attention operator, data parallelism can be determined as a parallel strategy; when the operator type is linear operator or convolution operator, tensor parallelism can be determined as a parallel strategy; when the operator type is a large model deep structure, pipeline parallelism can be determined as a parallel strategy; and when the operator type is a large image processing operator, tensor parallelism can be determined as a parallel strategy.

[0050] In this invention, when data stream analysis identifies an operator type as an attention operator, a data parallelism strategy, i.e., the dynamic programming (DP) strategy, is triggered. In some implementations, the batch data of the input subtask is divided into multiple sub-batches, and complete copies of the attention operator are deployed on multiple computing cards. Each computing card independently processes its corresponding sub-batch data, leveraging the inherent compatibility of the attention mechanism with data parallelism—meaning that the computations of each card are independent of each other—to achieve request-level concurrent processing.

[0051] When the operator type is identified as a linear operator (such as a fully connected layer) or a convolutional operator, a tensor parallel strategy, i.e., the TP strategy, will be adopted. In some implementations, the weight matrix or convolutional kernel of the linear operator can be split onto different computing cards based on the principle of matrix partitioning, such as row partitioning or column partitioning. Finally, the results are merged through communication operations, which effectively reduces the memory usage of a single card (especially the weight memory) and solves the memory overflow problem caused by the excessive computation of a single operator.

[0052] When an operator is identified as belonging to a deep structure of a large model, a pipelined parallel strategy, or PP strategy, will be initiated. In some implementations, the model layers are divided into different compute cards according to their depth. Data flows sequentially through each card like a pipeline, transmitting intermediate activation values ​​through inserted communication nodes.

[0053] When the operator type is identified as a large image processing operator, such as convolution of high-resolution images, a tensor parallel strategy, i.e., the TP strategy, is also adopted. In some implementations, the image data is segmented according to the spatial or channel dimensions to effectively solve the memory bottleneck caused by the large size of a single image. At the same time, the data dependency of the image block boundary is handled by the communication logic automatically inserted by the compiler.

[0054] This invention achieves intelligent decision-making at compile time by strongly binding operator characteristics with parallel strategies. This strategy selection method avoids the trial-and-error costs of blindly trying different parallel strategies in traditional methods, ensuring that the compiled executable file achieves theoretically optimal performance on a specific hardware topology. For special scenarios such as large image processing, it forces the use of tensor parallelism, effectively solving the problem of insufficient video memory in traditional data parallelism when processing high-resolution input.

[0055] According to some embodiments, the computing task graph can be divided into multiple subtasks based on the number of computing cards in the hardware parameters; and the multiple subtasks can be assigned to the corresponding target computing cards according to the connection method between computing cards in the interconnection topology information.

[0056] This invention can first read the number of computing cards in the hardware parameters of the target multi-card system, and then divide the computing task graph into equal parts. The process involves creating multiple subtasks, which are then assigned to their respective target computing cards according to the connection order corresponding to the connection method between the computing cards. The interconnect topology information can be used to describe the physical connection method, communication bandwidth, and communication latency between the computing cards.

[0057] In some implementations, nodes (i.e., operators) in the task graph can be clustered according to a preset granularity (such as layers, operator groups, or subgraphs). Each logical unit. This partitioning is not a simple equal distribution, but rather combines the topology of the task graph (such as the critical path and node weights) to ensure that the partitioned result is a logical unit. Each logic unit is calculated to be as balanced in terms of computational load as possible, thereby generating There are 10 subtasks to be assigned.

[0058] Traditional task partitioning often relies on fixed configuration files or hard coding. This invention automates and dynamically adapts task partitioning by reading the number of computing cards in the hardware parameters, greatly enhancing the versatility of the compilation method and solving the problem of poor portability caused by differences in hardware scale in heterogeneous environments.

[0059] According to some embodiments, the partitioning dimension can be determined according to the parallel strategy in the unified intermediate representation layer, and the operators corresponding to multiple subtasks can be partitioned based on the partitioning dimension.

[0060] This invention maps the divided subtasks to a unified intermediate representation layer of the target multi-GPU system. In this layer, all operators are abstracted as a collection of computation kernels and tensor objects. Tensor objects contain their logical shape, data type, and storage attributes. This abstraction shields the differences in instruction sets between the underlying computing GPUs, providing a unified operational view for subsequent partitioning.

[0061] Read the parallel strategy determined in the previous steps and parse the corresponding partitioning dimension.

[0062] For tensor parallel strategies: if the strategy is specified as column partitioning, the partitioning dimension is determined as the output channel dimension of the weight matrix; if the strategy is specified as row partitioning, the partitioning dimension is determined as the input channel dimension of the weight matrix.

[0063] For the data parallel strategy: the partitioning dimension is determined to be the batch size dimension.

[0064] Based on the segmentation dimensions obtained above, specific tensor segmentation operations are performed on the operators in the subtasks, such as weight matrix segmentation, feature map segmentation, and spatial segmentation. Spatial segmentation, such as in large image processing, can be performed by cutting along the height or width axis.

[0065] New subtasks are generated after the unified intermediate representation layer. These subtasks contain scaled-down tensors and modified computational logic, ensuring that they can be adapted to the memory capacity of the target computing card.

[0066] This invention decentralizes the segmentation operation to a unified intermediate representation layer, making the segmentation process completely transparent to the original model code. Regardless of the deep learning framework the original model is based on, it can be processed uniformly at the IR layer. This not only significantly reduces the adaptation cost for developers but also ensures the standardization of the segmentation logic, avoiding inconsistencies caused by manual coding errors. This invention does not blindly segment tensors by arbitrary dimensions but strictly follows the parallel strategy generated by the preceding steps. This dimensionality control ensures that the computational logic after segmentation is mathematically equivalent to the original task, guaranteeing the correctness and effectiveness of the segmentation.

[0067] According to some embodiments, the physical path for cross-card data transmission can be determined based on the connection method in the interconnection topology information; at the boundary of the sub-task corresponding to the physical path, a communication primitive node matching the parallel strategy is inserted as a cross-card communication node to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding sub-task.

[0068] In this invention, after subtask partitioning and tensor partitioning are completed, the graph optimization stage begins. First, the connection methods in the interconnect topology information are deeply analyzed to construct a physical connection graph, which contains bandwidth and latency information between all target compute cards. Based on this physical connection graph, the physical path for cross-card data transmission is determined using a shortest path algorithm or a bandwidth-based routing algorithm. For example, if there is a direct NVLink (NVIDIA Link) connection between compute card 0 and compute card 1, and compute card 2 needs to communicate with them via a PCIe Switch (Peripheral Component Interconnect Express Switch), the compiler will record the high-bandwidth direct connection path between card 0 and card 1, as well as the potential bottleneck path when card 2 participates in the communication.

[0069] After determining the physical path, the corresponding communication primitive is matched according to the type of parallel strategy. In some implementations, for tensor parallel scenarios: if the subtask involves slice computation of matrix multiplication, such as needing to summarize the results after row partitioning, an All-Reduce communication primitive node is inserted at the boundary of the output subtask. This node is responsible for reducing the local results of each computation card on the physical path, such as summing, and broadcasting them synchronously. For pipelined parallel scenarios: if the subtask is located at the boundary of different micro-batches and needs to pass activation values ​​or gradients, the compiler will insert a Send / Recv communication primitive node at the end of the forward propagation subtask or the beginning of the backward propagation subtask to establish a point-to-point data transmission channel between computation cards. For data parallel scenarios: at the end of each training step, an All-Gather or Reduce Scatter node can be inserted for the gradient of the model weights to ensure the consistency of model parameters on each card.

[0070] These inserted communication primitive nodes will be converted into machine instructions supported by the underlying hardware, containing the complete execution flow of computational subtasks and communication nodes, packaged and compiled into executable files that can be uniformly loaded and executed on the target multi-GPU system, such as CUDA (Compute Unified Device Architecture) binaries or firmware images of specific AI chips.

[0071] This invention determines the physical path based on interconnect topology information, enabling data transmission to bypass low-bandwidth bottleneck links, reducing data waiting time for computing cards, and improving hardware utilization. By using policy-driven primitive selection, the mathematical correctness of distributed computing is ensured, allowing the generated executable file to strictly adhere to the designed parallel logic when running on multi-card systems. This avoids training divergence or inference failures caused by communication semantic errors, thus achieving a balance between high performance and high reliability.

[0072] According to some embodiments, at the subtask boundary, the communication type and data dependency relationship specified by the parallel strategy can be identified; according to the communication type, the corresponding communication primitive node is selected from a predefined communication primitive library, where the communication primitive nodes include set communication primitive nodes, point-to-point communication primitive nodes, and data redistribution primitive nodes; the selected communication primitive node is inserted into the subtask boundary, and the input-output tensor mapping relationship of the selected communication primitive node and the target computing card identifier set participating in the communication are configured; based on the subtask graph after inserting and configuring the communication primitive nodes, device-side executable code segments corresponding to multiple target computing cards are generated, and a unified global symbol table and cross-card synchronization point information are set; based on the device-side executable code segments, the unified global symbol table, and the cross-card synchronization point information, an executable file is generated so that the target computing card loads and executes the corresponding subtask.

[0073] In this invention, after dividing the computation graph according to the parallel strategy, the data flow relationship between adjacent subtasks is first analyzed at the subtask boundaries corresponding to the physical paths. In some implementations, the communication type and data dependency relationship specified by the parallel strategy are identified by traversing the edges between subtasks and checking the cross-card usage of tensors, such as producer-consumer relationships, data aggregation requirements, and the target card range for data distribution.

[0074] Based on the identified communication type, the corresponding communication primitive node is selected from a predefined communication primitive library. The communication primitive library contains at least three types of nodes: set communication primitive nodes (used for collective operations between multiple cards); point-to-point communication primitive nodes (used to specify Send / Recv operations between two compute cards); and data redistribution primitive nodes (used to transform tensors from one distribution state, such as sharding or copying, to another distribution state, such as global copying or shard rearrangement). In some implementations, if the parallel strategy requires all compute cards to sum and synchronize their respective computation results, the AllReduce primitive node is selected; if only the master card needs to distribute model parameters to the other cards, the Broadcast primitive node is selected.

[0075] Insert the selected communication primitive node at the appropriate position at the subtask boundary, such as between the computation graphs of two subtasks, or between the output of one subtask and the input of the next. Then, configure the following attributes of this node: input / output tensor mapping, and the set of target computation card identifiers participating in the communication. Input / output tensor mapping: Specifies which input tensors come from the computation results of which cards, and how the output tensors should be distributed to the card inputs of subsequent subtasks; Target computation card identifier set participating in the communication: Specifies which computation cards need to participate in this communication operation. Attributes such as communication data type, buffer size, and asynchronous / synchronous mode can also be configured.

[0076] After inserting and configuring all communication primitive nodes, a holistic subtask graph containing primitive computational subtasks and communication operations is obtained. Based on this graph, a corresponding device-side executable code segment is independently generated for each target computing card. Each code segment contains only the computational subtasks that the card needs to execute and the communication operations related to that card. Simultaneously, a unified global symbol table is generated, recording all cross-card visible symbols, such as tensor names, communication buffer addresses, and function entry points, ensuring that different cards can correctly reference the same semantic object when executing different code segments.

[0077] Configure cross-card synchronization point information, such as inserting explicit synchronization instructions at locations where communication completion is required. Encapsulate all the generated device-side executable code segments, a unified global symbol table, and cross-card synchronization point information into a single executable file that can be loaded as a whole and executed on a per-card basis on the target multi-card system. When this executable file is loaded into the target multi-card system, the driver or runtime system distributes the corresponding code segments to each compute card based on the header metadata, loads the global symbol table, and coordinates the execution flow of each card according to the synchronization point information. This enables multiple cards to collaboratively execute their respective subtasks and automatically trigger communication operations at preset boundaries.

[0078] This invention identifies the communication type and data dependencies specified by the parallel strategy at the subtask boundary, enabling precise determination of the triggering conditions and data flow for cross-card communication, thus avoiding redundant synchronization or data mismatch caused by blindly inserting communication operations. By selecting corresponding set communication, point-to-point communication, or data redistribution primitive nodes from a predefined communication primitive library based on the communication type, the invention achieves standardization and modularization of communication operations, allowing different hardware backends to reuse the same set of communication descriptions and reducing code generation complexity. By inserting selected communication primitive nodes into the boundaries of subtasks and configuring input / output tensor mapping relationships and the target computing card identifier set participating in communication, the data transformation rules and participation scope of each communication operation are clarified, thereby ensuring the correctness of data distribution and the controllability of communication scope during multi-card execution. By generating device-side executable code segments for each card based on the inserted primitive nodes and configured subtask graph, and setting a unified global symbol table and cross-card synchronization point information, each computing card generates independent, lightweight code containing only its required subtasks and communication operations. The global symbol table ensures consistent addressing of symbols across cards, while the synchronization point information explicitly characterizes the execution boundaries of inter-card collaboration. By generating executable files based on device-side executable code segments, a unified global symbol table, and cross-card synchronization point information, a single executable entity is ultimately formed that can be loaded as a whole on the target multi-card system and executed according to card allocation.

[0079] According to some embodiments, a first type of operator and a second type of operator can also be identified in the computation task graph. The first type of operator is an operator with a one-to-one input-output relationship, and the second type of operator is an operator with a one-to-many, many-to-one, or many-to-many input-output relationship. An operator group is constructed with each second type of operator as the starting node, and the computation task graph is traversed downwards using a greedy strategy according to the data flow direction. If the downstream operator encountered is a first type of operator, the downstream operator is included in the corresponding operator group. If the downstream operator encountered is a second type of operator or the graph boundary is reached, the traversal is stopped. Explicit arrangement operators used to adjust the data memory layout between adjacent computation operators within the operator group are eliminated, and the layout format of the second type of operator is integrated with the layout format of the first type of operator by uniformly planning the memory layout of the data flow within the group.

[0080] This invention first performs a depth-first or breadth-first traversal of the input computation task graph to parse the input-output tensor topology of each node (operator). The specific classification logic can be as follows: First-class operator identification (one-to-one): If an operator receives only a single input tensor and generates a single output tensor, and its output is used only as the input of the next operator, then it is marked as a first-class operator. This type of operator does not change the logical dimension of the data, only the numerical value.

[0081] Second-class operator identification (one-to-many / many-to-one / many-to-many): If the input-output relationship of an operator is complex, it is marked as a second-class operator. Specifically, this may include: 1. One-to-many: such as the Split operator, where one input is split into multiple outputs.

[0082] 2. Many-to-one: such as Concat (concatenation) or Add (element-by-element addition), multiple inputs are combined into one output.

[0083] 3. Many-to-many: such as complex transformations combining Reshape and transpose, or multi-head splitting and merging in the Attention mechanism.

[0084] For each Class II operator, create an independent operator group and use it as the root or starting node of that group. Starting from this node, perform a depth-first traversal along the data flow (from input to output), checking the downstream neighbor operators encountered. If the neighbor operator is a Class I operator (one-to-one), add it to the current operator group and continue traversing the downstream of that Class I operator (i.e., recursive or iterative processing). If the neighbor operator is a Class II operator (i.e., another complex operator is encountered), or the node is an output node of the computation graph (graph boundary), immediately stop traversing the current branch.

[0085] After constructing the operator groups, memory layout optimization can be performed on each operator group. Analyze the data memory layout requirements of all operators within the group. For example, some operators may require NHWC (batch size, height, width, number of channels) format, while others require NCHW format. By uniformly planning the data flow within the group, the final layout format required to compute the second type of operators (the core of the group) is determined. The equivalent computation logic of the first type of operators (simple operators within the group) under this layout is derived, or the index mapping for data reading / writing is directly adjusted. Finally, the logic of multiple operators within the operator group is merged into a single fusion kernel, physically removing the intermediate nodes originally used for adjusting memory placement.

[0086] This invention breaks away from the traditional limitation of fusing only consecutive element-wise operators by starting with a second-type operator of one-to-many / many-to-one type and employing a greedy strategy to absorb one-to-one first-type operators. This allows for the construction of larger and more complete computational units (operator groups). During execution, previously scattered multiple kernel calls are merged into a single call, significantly reducing hardware scheduling overhead and context switching time, and substantially improving computational density and pipeline utilization. Through mathematical equivalence transformations, first-type operators can directly operate on the layout format required by second-type operators, transforming the originally expensive data movement operations into lightweight logical index adjustments. This not only reduces the number of nodes in the computation graph but also substantially reduces memory access bandwidth requirements and intermediate cache memory usage, thereby significantly improving the overall throughput of model inference or training and reducing latency.

[0087] The following describes an embodiment of the apparatus of the present invention, which can be used to perform the embodiment of the method of the present invention. For details not disclosed in the embodiment of the apparatus of the present invention, please refer to the embodiment of the method of the present invention.

[0088] Figure 6 This is a block diagram of a graph-optimized compiler for multi-GPU heterogeneous inference provided in an embodiment of the present invention. Figure 6 As shown, the graph optimization compilation device 600 for multi-card heterogeneous inference includes an operator type determination module 601, a parallel strategy determination module 602, a task partitioning module 603, and a tensor partitioning module 604.

[0089] The operator type determination module 601 is used to obtain the hardware parameters, interconnection topology information and computing task graph of the target multi-card system, and to perform data flow analysis on the computing task graph to determine the operator type in the computing task graph. Parallel strategy determination module 602 is used to determine the parallel strategy based on the operator type; The task partitioning module 603 is used to partition the computing task graph according to hardware parameters and interconnect topology information to determine multiple sub-tasks and the target computing cards corresponding to each sub-task. Tensor segmentation module 604 is used to perform tensor segmentation on multiple subtasks according to interconnection topology information and parallel strategy in the unified intermediate representation layer of the target multi-card system, and insert cross-card communication nodes between multiple subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask.

[0090] Optionally, the parallel strategy determination module 602 is specifically used for: When the operator type is attention operator, data parallelism is determined as the parallel strategy; When the operator type is a linear operator or a convolution operator, tensor parallelism is determined as the parallel strategy; When the operator type is a large model with a deep structure, pipeline parallelism is determined as the parallel strategy; When the operator type is a large image processing operator, tensor parallelism is determined as the parallel strategy.

[0091] Optionally, the task splitting module 603 is specifically used for: Based on the number of computing cards in the hardware parameters, the computing task graph is divided into multiple subtasks; Based on the connection method between computing cards in the interconnection topology information, multiple subtasks are assigned to the corresponding target computing cards.

[0092] Optionally, the tensor partitioning module 604, in the unified intermediate representation layer of the target multi-card system, performs tensor partitioning on multiple subtasks according to interconnection topology information and parallel strategies, specifically for: In the unified intermediate representation layer, the partitioning dimension is determined according to the parallel strategy, and the operators corresponding to multiple subtasks are partitioned based on the partitioning dimension.

[0093] Optionally, the tensor segmentation module 604, in the case of inserting cross-card communication nodes between multiple subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask, is specifically used for: Based on the connection method in the interconnection topology information, determine the physical path for cross-card data transmission; At the boundary of the subtask corresponding to the physical path, a communication primitive node matching the parallel strategy is inserted as a cross-card communication node to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask.

[0094] Optionally, the tensor partitioning module 604 inserts communication primitive nodes matching the parallel strategy as cross-card communication nodes at the boundaries of subtasks corresponding to the physical path, in order to generate an executable file that is uniformly loaded and executed on the target multi-card system. Specifically, this is used when the target computing card loads and executes the corresponding subtasks. At the boundaries of subtasks, identify the communication types and data dependencies specified by the parallel strategy; Based on the communication type, select the corresponding communication primitive node from the predefined communication primitive library, where the communication primitive nodes include set communication primitive nodes, point-to-point communication primitive nodes, and data redistribution primitive nodes; Insert the selected communication primitive node into the boundary of the subtask, and configure the input-output tensor mapping relationship of the selected communication primitive node and the target computing card identifier set participating in the communication; Based on the subtask graph after inserting and configuring the communication primitive nodes, generate the device-side executable code segments corresponding to each of the multiple target computing cards, and set a unified global symbol table and cross-card synchronization point information; The executable file is generated based on the device-side executable code segment, the unified global symbol table, and the cross-card synchronization point information, so that the target computing card can load and execute the corresponding subtask.

[0095] Optionally, the graph-optimized compiler 600 for multi-GPU heterogeneous inference also includes an operator fusion module 605, used for: Identify a first type of operator and a second type of operator in the computation task graph, wherein the first type of operator is an operator whose input and output are one-to-one, and the second type of operator is an operator whose input and output are one-to-many, many-to-one, or many-to-many; A group of operators is constructed with each of the second type of operators as the starting node, and the computation task graph is traversed downwards using a greedy strategy according to the data flow direction. If the downstream operator encountered is of the first type, the downstream operator is included in the corresponding operator group. Stop traversing if the downstream operator encountered is a second type of operator or if the graph boundary is reached; Eliminate explicit layout operators used to adjust data memory layout between adjacent computation operators within an operator group, and integrate the layout format of the second type of operator with the layout format of the first type of operator by uniformly planning the memory layout of data flow within the group.

[0096] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0097] Figure 7 A schematic diagram of the structure of the electronic device provided in the embodiments of the present invention is shown below. Figure 7 As shown, the electronic device 700 of this embodiment may include a memory 701 and a processor 702.

[0098] The memory 701 stores a computer program, which, when executed by the processor 702, causes the processor 702 to perform the method described in the above embodiments.

[0099] The processor 702 and the memory 701 are connected, for example, via a bus.

[0100] Optionally, the electronic device 700 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 700 does not constitute a limitation on the embodiments of the present invention.

[0101] Processor 702 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 702 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0102] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0103] The memory 701 can be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD. ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0104] The memory 701 stores application code that executes the present invention, and its execution is controlled by the processor 702. The processor 702 executes the application code stored in the memory 701 to implement the content shown in the foregoing method embodiments.

[0105] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0106] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0107] The present invention also provides a non-transitory computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0108] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0109] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of the present invention, its specific implementation methods, and its application scope, are all within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A graph-optimized compilation method for multi-GPU heterogeneous inference, characterized in that, include: The hardware parameters, interconnect topology information, and computing task graph of the target multi-card system are obtained, and data flow analysis is performed on the computing task graph to determine the operator types in the computing task graph. Based on the operator type, determine the parallel strategy; Based on the hardware parameters and the interconnection topology information, the computing task graph is divided into tasks to determine multiple sub-tasks and the target computing cards corresponding to each of the multiple sub-tasks; In the unified intermediate representation layer of the target multi-card system, tensor partitioning is performed on each of the multiple subtasks according to the interconnection topology information and the parallel strategy, and cross-card communication nodes are inserted between the multiple subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask.

2. The method according to claim 1, characterized in that, The determination of the parallel strategy based on the operator type includes: When the operator type is an attention operator, data parallelism is determined as a parallel strategy; When the operator type is a linear operator or a convolution operator, tensor parallelism is determined as the parallel strategy; When the operator type is a large model with a deep structure, pipeline parallelism is determined as the parallel strategy; When the operator type is a large image processing operator, tensor parallelism is determined as the parallel strategy.

3. The method according to claim 1, characterized in that, The step of dividing the computing task graph into tasks based on the hardware parameters and the interconnection topology information to determine multiple subtasks and their respective target computing cards includes: Based on the number of computing cards in the hardware parameters, the computing task graph is divided into the multiple sub-tasks; According to the connection method between computing cards in the interconnection topology information, the multiple subtasks are assigned to the corresponding target computing cards.

4. The method according to claim 1, characterized in that, In the unified intermediate representation layer of the target multi-card system, tensor partitioning is performed on each of the multiple subtasks based on the interconnection topology information and the parallel strategy, including: In the unified intermediate representation layer, the segmentation dimension is determined according to the parallel strategy, and the operators corresponding to the multiple subtasks are segmented based on the segmentation dimension.

5. The method according to claim 1, characterized in that, The step of inserting cross-card communication nodes between the multiple subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtasks, includes: Based on the connection method in the interconnection topology information, determine the physical path for cross-card data transmission; At the boundary of the subtask corresponding to the physical path, a communication primitive node matching the parallel strategy is inserted as the cross-card communication node to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask.

6. The method according to claim 5, characterized in that, The step involves inserting a communication primitive node matching the parallel strategy at the boundary of the sub-task corresponding to the physical path as the cross-card communication node to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding sub-task, including: At the boundary of the subtask, identify the communication type and data dependency specified by the parallel strategy; According to the communication type, a corresponding communication primitive node is selected from a predefined communication primitive library, wherein the communication primitive node includes set communication primitive node, point-to-point communication primitive node and data redistribution primitive node; Insert the selected communication primitive node into the boundary of the subtask, and configure the input-output tensor mapping relationship of the selected communication primitive node and the target computing card identifier set participating in the communication; Based on the subtask graph after inserting and configuring the communication primitive nodes, generate the device-side executable code segments corresponding to each of the multiple target computing cards, and set a unified global symbol table and cross-card synchronization point information; The executable file is generated based on the device-side executable code segment, the unified global symbol table, and the cross-card synchronization point information, so that the target computing card can load and execute the corresponding subtask.

7. The method according to claim 1, characterized in that, Also includes: Identify a first type of operator and a second type of operator in the computation task graph, wherein the first type of operator is an operator whose input and output are one-to-one, and the second type of operator is an operator whose input and output are one-to-many, many-to-one, or many-to-many; A group of operators is constructed with each of the second type of operators as the starting node, and the computation task graph is traversed downwards using a greedy strategy according to the data flow direction. If the downstream operator encountered is of the first type, the downstream operator is included in the corresponding operator group. Stop traversing if the downstream operator encountered is a second type of operator or if the graph boundary is reached; Eliminate explicit layout operators used to adjust data memory layout between adjacent computation operators within an operator group, and integrate the layout format of the second type of operator with the layout format of the first type of operator by uniformly planning the memory layout of data flow within the group.

8. A graph-optimized compiler for multi-GPU heterogeneous inference, characterized in that, include: The operator type determination module is used to acquire the hardware parameters, interconnect topology information and computing task graph of the target multi-card system, and perform data flow analysis on the computing task graph to determine the operator type in the computing task graph; A parallel strategy determination module is used to determine a parallel strategy based on the operator type; The task segmentation module is used to segment the computing task graph according to the hardware parameters and the interconnection topology information to determine multiple sub-tasks and the target computing cards corresponding to each of the multiple sub-tasks; The tensor partitioning module is used to partition the multiple subtasks into tensors according to the interconnection topology information and the parallel strategy in the unified intermediate representation layer of the target multi-card system, and insert cross-card communication nodes between the multiple subtasks to generate an executable file that is uniformly loaded and executed on the target multi-card system, so that the target computing card loads and executes the corresponding subtask.

9. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images