Efficient collective communication method and device for heterogeneous GPU cluster

Abstract

The invention discloses a heterogeneous GPU cluster-oriented efficient collective communication method and device, and the method comprises the steps: obtaining a target communication operator set and a physical topology corresponding to a heterogeneous GPU cluster, and determining an initial state constraint and a target state constraint; the scheduling time cost of each communication scheme is determined, and the communication scheme corresponding to the minimum scheduling time cost in the multiple scheduling time costs is selected as the efficient collective communication mode of the heterogeneous GPU cluster. According to the method, the communication scheme corresponding to the minimum scheduling time cost of the heterogeneous GPU cluster is determined to serve as the efficient collective communication mode of the heterogeneous GPU cluster based on the actual conditions of topological structure difference, link bandwidth imbalance and the like in the heterogeneous GPU cluster environment; it is ensured that the communication process is not limited by the bandwidth bottleneck, and system resource utilization efficiency and parallel training performance are prevented from being limited.

Description

Technical Field

[0001] This invention relates to the field of communication technology for heterogeneous GPU clusters, and more particularly to an efficient collective communication method and apparatus for heterogeneous GPU clusters. Background Technology

[0002] As the scale of large language models and machine learning models continues to expand, the demands on computing power and communication performance for model training are constantly increasing. To meet this demand, large-scale data and model parameters are typically distributed across multiple servers, and training is completed collaboratively through various methods such as data parallelism, tensor parallelism, and pipelined parallelism. During parallel training, frequent collective communication operations are performed to synchronize model gradients, optimizer states, and model parameters; communication efficiency directly affects overall training performance.

[0003] Currently, collective communication, as a core communication operation in distributed parallel computing frameworks, is widely used in scenarios such as data-parallel training, model-parallel training, and scientific computing. Common collective communication operators include AllReduce, AllGather, ReduceScatter, and Broadcast, which efficiently coordinate data transmission among multiple computing nodes and have a decisive impact on the overall system performance. To support efficient large-scale distributed training, GPU manufacturers provide high-performance collective communication libraries such as NVIDIA's NCCL and AMD's RCCL. These libraries implement the scheduling and execution of common communication operators through pre-defined standard algorithms and have become the mainstream solution in current distributed training systems.

[0004] However, as the cluster size continues to expand and the interconnection topology between nodes becomes more complex, existing standard collective communication algorithms generally adopt a unified scheduling strategy, ignoring the actual situation of topology differences and uneven link bandwidth in heterogeneous GPU cluster environments. This makes the communication process susceptible to bandwidth bottlenecks, severely restricting the efficiency of system resource utilization and parallel training performance, and becoming a key bottleneck problem that hinders the scalability of large-scale model training. Summary of the Invention

[0005] Based on this, it is necessary to propose an efficient collective communication method, device, computer equipment, and storage medium for heterogeneous GPU clusters to address the above problems.

[0006] An efficient collective communication method for heterogeneous GPU clusters, the method comprising:

[0007] Obtain the target communication operator set corresponding to the heterogeneous GPU cluster, and determine the initial state constraints and target state constraints based on the target communication operator set;

[0008] The physical topology corresponding to the heterogeneous GPU cluster is obtained, and the physical topology is described according to the DSL rules to obtain the description result. The description result is then parsed by the built-in interpreter of the DSL rules to generate a directed graph model corresponding to the physical topology. The directed graph model consists of a set of GPU nodes and a set of physical topology interconnections. The set of GPU nodes consists of all GPU nodes in each heterogeneous GPU cluster node. The set of physical topology interconnections consists of a first interconnection relationship and a second interconnection relationship. The first interconnection relationship is the relationship between GPU nodes within the heterogeneous GPU cluster node. The second interconnection relationship is the relationship between nodes in the heterogeneous GPU cluster.

[0009] Configure link bandwidth and number of links on multiple physical topology interconnections;

[0010] Determine the minimum link bandwidth from the plurality of said link bandwidths;

[0011] The maximum number of parallel data blocks is determined based on the minimum link bandwidth, the link bandwidth, and the number of links;

[0012] When the GPU nodes are connected via shared links, the physical topology interconnection of multiple shared links constitutes a shared link group;

[0013] The rated bandwidth of the shared link is determined based on the link bandwidth and the number of links corresponding to the shared link;

[0014] By using the Z3 solver, a communication scheme that satisfies the constraints in the directed graph model is selected. The communication scheme is the order in which data is transmitted on the GPUs in the heterogeneous GPU cluster nodes. The communication scheme must satisfy the initial state constraints and the target state constraints of the target communication operator.

[0015] The scheduling time cost of each communication scheme is determined, and the communication scheme with the smallest scheduling time cost among multiple schemes is selected as the efficient collective communication method for the heterogeneous GPU cluster.

[0016] In one embodiment, the physical topology includes the following parameters: the total number of heterogeneous GPU cluster nodes, the number of GPU nodes within the heterogeneous GPU cluster nodes, the first interconnection relationship between GPU nodes within the heterogeneous GPU cluster nodes, and the second interconnection relationship between heterogeneous GPU cluster nodes.

[0017] In one embodiment, the parameters included in the physical topology are specifically:

[0018] n

[0019] in, This represents the total number of nodes in the heterogeneous GPU cluster. When inputting, the total number of nodes in the heterogeneous GPU cluster is specified as n.

[0020]

[0021] in, This represents the number of GPUs in the i-th heterogeneous GPU cluster node. The number of GPUs in the heterogeneous GPU cluster node is specified as the input. ;

[0022]

[0023] in, This represents the first interconnection relationship between GPU nodes within node i of a heterogeneous GPU cluster; Indicates the link type; the link type includes direct interconnection. Interconnect with switches ; Indicates the source GPU's identifier; Indicates the target GPU's ID; This indicates the link bandwidth; the unit is GB / s. Indicates the number of links, which is a positive integer;

[0024]

[0025] in, This indicates the second interconnection relationship between node i and node j in the heterogeneous GPU cluster, and configures the link type between the nodes in the heterogeneous GPU cluster. , For GPUs communicating with each other from node i in a heterogeneous GPU cluster, For the GPUs outward from node j in the heterogeneous GPU cluster, This indicates the link bandwidth, measured in GB / s. This represents the number of links and is a positive integer.

[0026] In one embodiment, the initial state constraint and the target state constraint are as follows:

[0027] Initial state constraints:

[0028]

[0029] Target state constraints:

[0030]

[0031] in, A collection of data blocks; Initial state constraints; For the g-th GPU; Constraints for the target state; For the first data block in the set indivual ; For the first data block in the set One data block.

[0032] In one embodiment, the constraint includes:

[0033] Transmission rule constraint: When a data block is sent from the source GPU to the target GPU, the sending time should be earlier than the receiving time, that is, the time when the data block appears on the source GPU is less than the time when the data block appears on the target GPU.

[0034]

[0035] in, For source GPU, For the target GPU, For the first data block in the set indivual ; For the first indivual The time it appeared on the source GPU, For the first indivual The time of appearance on the target GPU;

[0036] Direct link bandwidth constraint: No data blocks exceeding the bandwidth limit can be sent simultaneously on the same link;

[0037]

[0038] in, Indicates time The first in the inner data block set indivual Send from the source GPU to the target GPU;

[0039] Shared link bandwidth constraint: The sum of the bandwidths of multiple links in each shared link group R cannot exceed the rated bandwidth of the shared link group R;

[0040]

[0041] in, This is the rated bandwidth; For the first data block in the set indivual .

[0042] In one embodiment, the minimum link bandwidth is achieved by the following expression:

[0043]

[0044] in, Minimum link bandwidth; Link bandwidth; It is a set of physical topology interconnections, where u and v are two GPU nodes or two heterogeneous GPU cluster nodes.

[0045] In one embodiment, the maximum number of parallel data blocks is achieved by the following expression:

[0046]

[0047] in, This represents the maximum number of parallel data blocks. Link bandwidth; This refers to the number of links; This represents the minimum link bandwidth.

[0048] In one embodiment, the rated bandwidth is achieved by the following expression:

[0049]

[0050] in, The rated bandwidth of the shared link; For shared link bandwidth; The number of links in the shared link; and All are GPU nodes in a shared blockchain; For shared link groups.

[0051] In one embodiment, the scheduling time cost of each communication scheme is determined by the following expression:

[0052]

[0053]

[0054] in, The time cost of scheduling a data block in the communication scheme that satisfies the constraints; The amount of data represented by the data block. Due to communication delay, It is the reciprocal of the link bandwidth. For data transmission time; It represents the reciprocal of the GPU's computational throughput. This refers to the data processing time on the GPU. Indicates the cost of time; This indicates the total number of scheduling steps.

[0055] A high-efficiency collective communication device for heterogeneous GPU clusters, comprising:

[0056] The system input module is used to obtain the target communication operator set corresponding to the heterogeneous GPU cluster, and determine the initial state constraints and target state constraints based on the target communication operator set; obtain the physical topology corresponding to the heterogeneous GPU cluster, describe the physical topology according to the DSL rules to obtain the description result, and parse the description result through the built-in interpreter of the DSL rules to generate a directed graph model corresponding to the physical topology; the directed graph model consists of a GPU node set and a physical topology interconnection relationship set; the GPU node set consists of all GPU nodes in each heterogeneous GPU cluster node; the physical topology interconnection relationship set consists of a first interconnection relationship and a second interconnection relationship, wherein the first interconnection relationship is the relationship between GPU nodes within the heterogeneous GPU cluster node; and the second interconnection relationship is the relationship between heterogeneous GPU cluster nodes.

[0057] The synthesis solver module is used to set link bandwidth and link quantity on multiple physical topology interconnections; determine the minimum link bandwidth from multiple link bandwidths; determine the maximum number of parallel data blocks based on the minimum link bandwidth, the link bandwidth, and the link quantity; when the GPU nodes are connected through shared links, the physical topology interconnections formed by multiple shared links constitute a shared link group; determine the rated bandwidth of the shared link based on the link bandwidth and link quantity corresponding to the shared link; select a communication scheme that satisfies the constraints in the directed graph model using the Z3 solver, wherein the communication scheme is the data transmission order on the GPUs in the heterogeneous GPU cluster nodes; the communication scheme must satisfy the initial state constraints and the target state constraints of the target communication operator;

[0058] The system output module determines the scheduling time cost of each communication scheme and selects the communication scheme with the smallest scheduling time cost among multiple schemes as the efficient collective communication method for the heterogeneous GPU cluster.

[0059] This application, based on the actual situation of topological differences and uneven link bandwidth in heterogeneous GPU cluster environments, determines the communication scheme corresponding to the minimum scheduling time cost of heterogeneous GPU clusters as an efficient collective communication method for heterogeneous GPU clusters; ensuring that the communication process is not limited by bandwidth bottlenecks, and avoiding limitations on system resource utilization efficiency and parallel training performance. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] in:

[0062] Figure 1 This is an application environment diagram of an efficient collective communication method for heterogeneous GPU clusters in one embodiment;

[0063] Figure 2 This is a flowchart of an efficient collective communication method for heterogeneous GPU clusters in one embodiment;

[0064] Figure 3 This is a schematic diagram of a heterogeneous GPU cluster node in one embodiment;

[0065] Figure 4 Here is an example diagram describing DSL rules in one embodiment;

[0066] Figure 5 This is a schematic diagram of a directed graph in one embodiment;

[0067] Figure 6 This is a performance comparison chart of the present invention and the prior art in terms of bandwidth utilization in one embodiment;

[0068] Figure 7 This is a comparison chart of the time performance of the present invention and the prior art in one embodiment;

[0069] Figure 8 This is a block diagram of a high-efficiency collective communication device for heterogeneous GPU clusters in one embodiment;

[0070] Figure 9 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] As the scale of large language models and machine learning models continues to expand, the demands on computing power and communication performance for model training are constantly increasing. To meet this demand, large-scale data and model parameters are typically distributed across multiple servers, and training is completed collaboratively through various methods such as data parallelism, tensor parallelism, and pipelined parallelism. During parallel training, collective communication operations are frequently executed to synchronize model gradients, optimizer states, and model parameters; communication efficiency directly affects overall training performance. Currently, collective communication, as a core communication operation in distributed parallel computing frameworks, is widely used in scenarios such as data-parallel training, model-parallel training, and scientific computing. Common collective communication operators include AllReduce, AllGather, ReduceScatter, and Broadcast, which efficiently coordinate data transmission between multiple computing nodes and have a decisive impact on overall system performance. To support efficient large-scale distributed training, GPU manufacturers provide high-performance collective communication libraries such as NVIDIA's NCCL and AMD's RCCL. These libraries implement the scheduling and execution of common communication operators through pre-defined standard algorithms and have become the mainstream solution in current distributed training systems. However, as cluster sizes continue to expand and inter-node interconnection topologies become increasingly complex, existing standard collective communication algorithms generally employ a uniform scheduling strategy, neglecting the realities of topological differences and uneven link bandwidth in heterogeneous GPU cluster environments. This makes the communication process susceptible to bandwidth bottlenecks, severely restricting system resource utilization efficiency and parallel training performance, becoming a key bottleneck hindering the scalability of large-scale model training.

[0073] To address the aforementioned technical issues, this application provides an efficient collective communication method for heterogeneous GPU clusters.

[0074] Figure 1 This is a diagram illustrating the application environment of an efficient collective communication method for heterogeneous GPU clusters in one embodiment. (Refer to...) Figure 1This efficient collective communication method for heterogeneous GPU clusters is applied to a system for efficient collective communication of heterogeneous GPU clusters. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, tablet computer, or laptop computer. The server 120 can be a standalone server or a server cluster consisting of multiple servers. Terminal 110 is used to obtain the target communication operator set corresponding to the heterogeneous GPU cluster, and determine the initial state constraints and target state constraints based on the target communication operator set; server 120 is used to obtain the physical topology corresponding to the heterogeneous GPU cluster, describe the physical topology according to the DSL rules to obtain the description result, and parse the description result through the built-in interpreter of the DSL rules to generate a directed graph model corresponding to the physical topology; the directed graph model consists of a GPU node set and a physical topology interconnection set; the GPU node set consists of all GPU nodes in each heterogeneous GPU cluster node; the physical topology interconnection set consists of a first interconnection relationship and a second interconnection relationship, the first interconnection relationship being the relationship between GPU nodes within the heterogeneous GPU cluster node; the second interconnection relationship being the relationship between heterogeneous GPU cluster nodes; in multiple physical topology interconnections The process involves setting link bandwidth and the number of links; determining the minimum link bandwidth from multiple link bandwidths; determining the maximum number of parallel data blocks based on the minimum link bandwidth, the link bandwidth, and the number of links; when GPU nodes are connected via shared links, the physical topology interconnection of multiple shared links constitutes a shared link group; determining the rated bandwidth of the shared link based on the link bandwidth and the number of links corresponding to the shared link; selecting a communication scheme that satisfies the constraints in the directed graph model using the Z3 solver, where the communication scheme is the data transmission order on the GPUs in the heterogeneous GPU cluster nodes; the communication scheme must satisfy the initial state constraints and the target state constraints of the target communication operator; determining the scheduling time cost of each communication scheme; and selecting the communication scheme with the minimum scheduling time cost among multiple schemes as the efficient collective communication method for the heterogeneous GPU cluster.

[0075] like Figure 2 As shown, in one embodiment, an efficient collective communication method for heterogeneous GPU clusters is provided. This method can be applied to both terminals and servers; this embodiment illustrates its application to terminals. The efficient collective communication method for heterogeneous GPU clusters specifically includes the following steps:

[0076] S1: Obtain the target communication operator set corresponding to the heterogeneous GPU cluster, and determine the initial state constraints and target state constraints based on the target communication operator set;

[0077] S2: Obtain the physical topology corresponding to the heterogeneous GPU cluster, describe the physical topology according to the DSL rules to obtain the description result, and parse the description result through the built-in interpreter of the DSL rules to generate a directed graph model corresponding to the physical topology; the directed graph model consists of a set of GPU nodes and a set of physical topology interconnections; the set of GPU nodes consists of all GPU nodes in each heterogeneous GPU cluster node; the set of physical topology interconnections consists of a first interconnection relationship and a second interconnection relationship, wherein the first interconnection relationship is the relationship between GPU nodes within the heterogeneous GPU cluster node; and the second interconnection relationship is the relationship between nodes in the heterogeneous GPU cluster.

[0078] S3: Set the link bandwidth and number of links on multiple physical topology interconnections;

[0079] S4: Determine the minimum link bandwidth from the plurality of said link bandwidths;

[0080] S5: Determine the maximum number of parallel data blocks based on the minimum link bandwidth, the link bandwidth, and the number of links;

[0081] S6: When the GPU nodes are connected through shared links, the physical topology interconnection relationship formed by multiple shared links constitutes a shared link group;

[0082] S7: Determine the rated bandwidth of the shared link based on the link bandwidth and the number of links corresponding to the shared link;

[0083] S8: Using the Z3 solver, a communication scheme that satisfies the constraints is selected in the directed graph model. The communication scheme is the order in which data is transmitted on the GPUs in the heterogeneous GPU cluster nodes. The communication scheme must satisfy the initial state constraints and the target state constraints of the target communication operator.

[0084] S9: Determine the scheduling time cost of each communication scheme, and select the communication scheme with the smallest scheduling time cost among multiple schemes as the efficient collective communication method for the heterogeneous GPU cluster.

[0085] Specifically, the physical topology used in this invention describes the actual hardware connection structure between computing devices in the system. It is the underlying foundation of the network architecture and is used to clarify the physical link relationships between each device. As shown in the figure below, the internal structure of a machine is illustrated, including eight graphics processing units (GPUs) connected by PCIe and NVLink links.

[0086] In one embodiment, the physical topology includes the following parameters: the total number of heterogeneous GPU cluster nodes, the number of GPU nodes within the heterogeneous GPU cluster nodes, the first interconnection relationship between GPU nodes within the heterogeneous GPU cluster nodes, and the second interconnection relationship between heterogeneous GPU cluster nodes.

[0087] In one embodiment, physical topology is required as one of the input information. To enable the system to accurately identify the topology and facilitate user description, a unified topology representation method is needed to support automated processing of subsequent communication path analysis and algorithm synthesis. This invention performs fine-grained modeling of the physical topology of heterogeneous GPU clusters. This modeling process includes, but is not limited to, the following: key parameters such as inter-node link connections, link bandwidth values, interconnection methods, link types, and collective communication scale. The parameters included in the physical topology are specifically as follows:

[0088] n(1)

[0089] in, This represents the total number of nodes in the heterogeneous GPU cluster. When inputting, the total number of nodes in the heterogeneous GPU cluster is specified as n.

[0090] (2)

[0091] in, This represents the number of GPUs in the i-th heterogeneous GPU cluster node. The number of GPUs in the heterogeneous GPU cluster node is specified as the input. ;

[0092] (3)

[0093] in, This represents the first interconnection relationship between GPU nodes within node i of a heterogeneous GPU cluster; Indicates the link type; the link type includes direct interconnection. Interconnect with switches ; Indicates the source GPU's identifier; Indicates the target GPU's ID; This indicates the link bandwidth; the unit is GB / s. Indicates the number of links, which is a positive integer;

[0094] (4)

[0095] in, This indicates the second interconnection relationship between node i and node j in the heterogeneous GPU cluster, and configures the link type between the nodes in the heterogeneous GPU cluster. , For GPUs communicating with each other from node i in a heterogeneous GPU cluster, For the GPUs outward from node j in the heterogeneous GPU cluster, This indicates the link bandwidth, measured in GB / s. Represents the number of links, which is a positive integer. Figure 3 The above is a schematic diagram of a heterogeneous GPU cluster node.

[0096] In one embodiment, the initial state constraint and the target state constraint are as follows:

[0097] Initial state constraints:

[0098] (5)

[0099] Target state constraints:

[0100] (6)

[0101] in, A collection of data blocks; Initial state constraints; For the g-th GPU; Constraints for the target state; For the first data block in the set indivual ; For the first data block in the set One data block.

[0102] In one embodiment, the constraint includes:

[0103] Transmission rule constraint: When a data block is sent from the source GPU to the target GPU, the sending time should be earlier than the receiving time, that is, the time when the data block appears on the source GPU is less than the time when the data block appears on the target GPU.

[0104]

[0105] in, For source GPU, For the target GPU, For the first data block in the set indivual ; For the first indivual The time it appeared on the source GPU, For the first indivual The time of appearance on the target GPU;

[0106] Direct link bandwidth constraint: No data blocks exceeding the bandwidth limit can be sent simultaneously on the same link;

[0107] (7)

[0108] in, Indicates time The first in the inner data block set indivual Send from the source GPU to the target GPU;

[0109] Shared link bandwidth constraint: The sum of the bandwidths of multiple links in each shared link group R cannot exceed the rated bandwidth of the shared link group R;

[0110] (8)

[0111] in, This is the rated bandwidth; For the first data block in the set indivual .

[0112] In one embodiment, the minimum link bandwidth is achieved by the following expression:

[0113] (9)

[0114] in, Minimum link bandwidth; Link bandwidth; It is a set of physical topology interconnections, where u and v are two GPU nodes or two heterogeneous GPU cluster nodes.

[0115] In one embodiment, the maximum number of parallel data blocks is achieved by the following expression:

[0116] (10)

[0117] in, This represents the maximum number of parallel data blocks. Link bandwidth; This refers to the number of links; This represents the minimum link bandwidth.

[0118] In one embodiment, targeting When connecting shared links such as switches and NVSwich, this invention uses the same quantization method, marking the edge weights of shared links as -1 to distinguish them from point-to-point links. Simultaneously, the edge e formed by all shared links involved in each switch... iThese links are grouped into a shared link group R to track sharing relationships. This method models the entire topology as point-to-point links, expressing different link types, diverse connection relationships, and dynamic bandwidth characteristics in a fine-grained manner. The edges in the directed graph model... Configure the following properties:

[0119] The rated bandwidth is achieved by the following expression:

[0120] (11)

[0121] in, The rated bandwidth of the shared link; For shared link bandwidth; The number of links in the shared link; and All are GPU nodes in a shared blockchain; For shared link groups.

[0122] In one embodiment, the evaluation method for scheduling time cost (i.e., the setting of the objective function) comprehensively considers three parts of the overhead in the aggregate communication process: communication latency (startup time), data transmission time, and GPU-side data processing time. Communication latency reflects the startup overhead of message transmission; data transmission time is determined by both link bandwidth and message size; and GPU computation time depends on the processing logic of the aggregate communication operator and the computing power of the participating nodes. The solver weights and combines these three factors into a total communication scheduling time cost, which is then minimized as the objective function.

[0123] The scheduling time cost for each communication scheme is determined by the following expression:

[0124] (12)

[0125] (9)

[0126] in, The time cost of scheduling a data block in the communication scheme that satisfies the constraints; The amount of data represented by the data block. Due to communication delay, It is the reciprocal of the link bandwidth. For data transmission time; It represents the reciprocal of the GPU's computational throughput. This refers to the data processing time on the GPU. Indicates the cost of time; Indicates the total number of scheduling steps. For example... Figure 6 and Figure 7The figures shown are a performance comparison chart of the present invention and the prior art in terms of bandwidth utilization and time performance, respectively.

[0127] Based on an in-depth analysis of existing collective communication scheduling methods, this invention has the following beneficial effects:

[0128] (1) This invention can uniformly abstract and express the heterogeneous characteristics such as bandwidth, latency, and topology between different GPUs, links and switching nodes, realize the information modeling of the heterogeneity of resources in heterogeneous GPU clusters, and provide a unified and standardized description basis for subsequent scheduling optimization.

[0129] (2) This invention proposes to quantify collective communication data traffic at the block level according to the bandwidth-delay cost model, divide the communication task into several standardized data blocks, refine the communication granularity, and explicitly represent the bandwidth consumption and transmission delay of the data blocks on heterogeneous links, thereby realizing bandwidth awareness and link load balancing, and ensuring that the scheduling process has accurate adaptation capabilities to the characteristics of heterogeneous links.

[0130] (3) This invention innovatively models the collective communication scheduling problem as an SMT solution problem. By constructing formal expressions for bandwidth, latency, reception constraints and thread block scheduling constraints, it uniformly describes the scheduling objectives and constraints, significantly improving the solvability and efficiency of the scheduling strategy. It is applicable to large-scale heterogeneous GPU cluster environments.

[0131] (4) The present invention adopts the counterexample-guided inductive synthesis method to dynamically detect counterexamples that do not meet the performance indicators or violate the constraints generated during the solution process, and transforms them into supplementary constraint increments added to the SMT model. Through the iterative feedback optimization mechanism, the search space is gradually converged, the solution speed is improved, and the performance superiority and consistency of the final scheduling scheme in the heterogeneous GPU cluster environment are guaranteed.

[0132] In summary, the collective communication scheduling method proposed in this invention fully considers the complexity of heterogeneous resource environments and the efficiency of scheduling optimization. It can significantly reduce scheduling solution time and improve communication performance, and has excellent scalability, practicality and application prospects.

[0133] To address the issue of excessively long algorithm synthesis times in large-scale heterogeneous GPU clusters, this invention integrates a counterexample-guided inductive synthesis method during the synthesis process, improving solution efficiency through a dynamic feedback optimization mechanism. Specifically, during automatic synthesis, the system identifies intermediate solutions that do not meet performance targets or violate constraints as counterexamples in real time, and analyzes and processes these counterexamples. The identified counterexamples are automatically transformed into new supplementary constraints and incrementally added to the existing SMT model for dynamic correction and shrinking of the solution space. This process continuously optimizes in an iterative manner, gradually guiding the solver towards a solution space that satisfies all constraints and has optimal performance.

[0134] The specific process is as follows:

[0135] 1. Dynamic detection and transformation of counterexamples

[0136] During the solution process, for the initial solution Or the current iterative solution Detects counterexamples that violate bandwidth limits and receive constraints. These counterexamples are automatically converted into supplementary constraints and incrementally added to the SMT model to dynamically correct and shrink the solution space.

[0137] 2. Specific details of the counterexample constraint

[0138] The constraints in the counterexample set C are transformed into a form that the SMT solver can recognize and added to the search space.

[0139] The main constraints include:

[0140] (1) Bandwidth constraint: For any time T, the number of data blocks transmitted simultaneously on any link must not exceed the bandwidth capacity of that link, to prevent bandwidth overflow and link congestion. Formal expression:

[0141] (13)

[0142] (2) Reception Constraint: Ensure that each data block is received only once on the target GPU to avoid duplication. Formal expression:

[0143] (14)

[0144] 3. Iterative search process guided by counterexamples

[0145] Counterexample constraints are gradually added to the SMT solver to guide subsequent search space reduction and optimization. The solver is based on the current solution. A constrained search is performed on the set of counterexamples C to generate a new solution M. If M satisfies all constraints, it is recorded as a new intermediate solution. As input for the next iteration. To further improve search efficiency, a guidance strategy based on recorded intermediate solutions is introduced. Specifically, this is achieved by maintaining the operation sequence of GPU thread block scheduling and, based on the steps of counterexamples. Set constraints for send and receive operations:

[0146] When iteration steps At the same time, the constraints remain consistent with previously recorded operations;

[0147] when At that time, adjust operations based on new counterexamples;

[0148] when At the same time, restrict operations that are not affected by counterexamples to ensure search convergence.

[0149] in, For iterative steps; The steps in the counterexample .

[0150] 4. Iterate alternately until convergence.

[0151] The search and verification phases are performed alternately. After a counterexample is detected during the verification phase, the constraint set and solution are updated, and this process is repeated iteratively until no counterexamples are generated, confirming the final solution. .

[0152] A high-efficiency collective communication device for heterogeneous GPU clusters, such as Figure 8 As shown, it includes:

[0153] The system input module 10 is used to obtain the target communication operator set corresponding to the heterogeneous GPU cluster, and determine the initial state constraints and target state constraints based on the target communication operator set; obtain the physical topology corresponding to the heterogeneous GPU cluster, describe the physical topology according to the DSL rules to obtain the description result, and parse the description result through the built-in interpreter of the DSL rules to generate a directed graph corresponding to the physical topology; the directed graph consists of a GPU node set and a physical topology interconnection set; the GPU node set consists of all GPU nodes in each heterogeneous GPU cluster node; the physical topology interconnection set consists of a first interconnection relationship and a second interconnection relationship, wherein the first interconnection relationship is the relationship between GPU nodes within the heterogeneous GPU cluster node; and the second interconnection relationship is the relationship between heterogeneous GPU cluster nodes.

[0154] The synthesis solver module 20 is used to set the link bandwidth and number of links on multiple physical topology interconnections; determine the minimum link bandwidth from multiple link bandwidths; determine the maximum number of parallel data blocks based on the minimum link bandwidth, the link bandwidth, and the number of links; when the GPU nodes are connected through shared links, the physical topology interconnections formed by multiple shared links constitute a shared link group; determine the rated bandwidth of the shared link based on the link bandwidth and the number of links corresponding to the shared link; and select a communication scheme that satisfies the constraints in the directed graph model using the Z3 solver, wherein the communication scheme is the data transmission order on the GPUs in the heterogeneous GPU cluster nodes; the communication scheme must satisfy the initial state constraints and the target state constraints of the target communication operator.

[0155] The system output module 30 determines the scheduling time cost of each communication scheme and selects the communication scheme with the smallest scheduling time cost among multiple schemes as the efficient collective communication method for the heterogeneous GPU cluster.

[0156] This embodiment employs a collective communication optimization method based on counterexample-guided inductive synthesis (CEGIS). The system describes the heterogeneous GPU cluster topology using a Domain-Specific Language (DSL). Next, the DSL rules are transformed into an Abstract Syntax Tree (AST), and a directed graph topology model quantifying link bandwidth is constructed based on the AST. Subsequently, based on the topology model, the data block routing problem is formally modeled as a Satisfiable Module Theory (SMT) formula, and corresponding formal constraints are set. The SMT solver is then invoked to perform synthesis. To improve solution efficiency, the system uses a counterexample-guided inductive synthesis (CEGIS) method. During the solution process, counterexamples that do not meet the constraints are dynamically detected, and new constraints are incrementally generated based on these counterexamples and added to the SMT model. This dynamically corrects and shrinks the solution space, continuously optimizing the solution process. Finally, the optimal collective communication algorithm obtained through synthesis is deployed to the GPU hardware platform via a collective communication backend, enabling efficient collective communication operations in a heterogeneous GPU cluster environment.

[0157] The input topology used in this example is as follows: Figure 2As shown, the experimental environment is based on an NVIDIA V100 16GB GPU heterogeneous GPU cluster. This topology is a typical heterogeneous configuration, including one server node equipped with 4 GPUs and another server node equipped with 8 GPUs. The 8-GPU node integrates two SXM2 modules, each GPU is equipped with 6 NVLink interfaces, and the single-port bandwidth is 25GB / s. The modules are interconnected via 48GB / s UPI links, and each pair of GPUs shares a 16GB / s PCIe switch. Furthermore, the 8-GPU server node has a network bandwidth of 100 Gbps, while the 4-GPU server nodes each have 64 Gbps depending on their configuration, forming a multi-node heterogeneous communication environment.

[0158] To address the communication problem of heterogeneous topology sets mentioned above, this embodiment includes the following steps.

[0159] Step 1: Physical topology modeling of heterogeneous GPU clusters:

[0160] The physical topology is described using the DSL language as input, and a directed graph model G is automatically generated through parsing. This serves as the basis for modeling and solving collective communication scheduling optimization. A specific example of a DSL rule description is shown below. Figure 4 As shown.

[0161] The configuration in the diagram describes a system consisting of 2 compute nodes ( The physical topology of a heterogeneous GPU cluster consisting of (2) components. The node contains 8 GPUs. Each node contains four GPUs, and the bandwidth and latency parameters for both intra-node and inter-node GPU devices are defined. Within a node, GPUs are interconnected via both switches and direct point-to-point connections, with each link specifying its bandwidth (in GB / s) and latency (in µs). Furthermore, the configuration defines the cross-node bandwidth and latency steps for inter-node GPU device interconnection via switches.

[0162] Step 2: Link bandwidth blockization:

[0163] Statistical analysis was performed on the link bandwidth in the heterogeneous GPU cluster topology, and the bandwidth was discretized into quantized bandwidth intervals in units of "blocks" based on the bandwidth size. The quantized directed graph is shown below. Figure 5 As shown.

[0164] This demonstrates a baseline with a minimum bandwidth of 5 GB / s. Each link can transmit one data block within one communication step. If there are n links between node u and node v, then the capacity of that link is defined as... This topology integrates both NVLINK and PCIe connections, covering both direct and shared links. Some nodes are interconnected via dual NVLINK links, achieving a total bandwidth of 50GB / s, while other nodes communicate via a 16GB / s PCIe shared link. Communication between nodes (different servers) relies on network interface cards (NICs), with varying link bandwidths of 5GB and 12.5GB / s respectively. The diagram clearly indicates the transmission capacity (bandwidth quantification) of each link and identifies the shared link group to which it belongs.

[0165] Step 3: Formal Modeling:

[0166] Transform each edge and vertex in the model into a formal constraint formula.

[0167] by Figure 5 Point-to-point For example, illustrate the constraints of this edge;

[0168] (15)

[0169] Passing at the same time The total communication traffic cannot exceed 10.

[0170] (16)

[0171] by Figure 5 Taking a shared edge group (category:2) as an example, we will explain the constraint definition corresponding to the shared edge group.

[0172] The total bandwidth of all edges with the category attribute of 2 at the same time is less than 3.

[0173] Step 4: Automatic Synthesis and Solution:

[0174] The specific algorithm for data aggregation is determined by using the set communication algorithm type as a hyperparameter. In Example 1, this hyperparameter is set to Allgather. The Allgather algorithm gradually aggregates the data from all nodes by exchanging and merging the data blocks held by each node with other nodes in stages, ultimately enabling each node to obtain the complete set of all data. For example, in a system with four nodes, nodes 1, 2, 3, and 4 each possess a portion of the data. After multiple rounds of exchange, all nodes can obtain the complete set of data from the four portions of 1, 2, 3, and 4, achieving efficient sharing of all data.

[0175] SMT uses the Allgather algorithm to determine the start and end states, and incorporates them along with the objective function into the model, which is then solved by the SMT solver. The constructed network includes a residual network, an attention module, a feature fusion module, a detection head, and a pedestrian coordinate calculation module.

[0176] Step 5: Counterexample Guidance: The CEGIS algorithm for generating a full aggregation algorithm with block = 1 on a 4+8 topology. First, an initial solution found using a greedy algorithm is captured. Next, a verification phase is performed where bandwidth and receive constraints are checked, and the first counterexample is identified (e.g., bandwidth congestion between GPU 1 and GPU 4 in step 1, and multiple receptions from GPU 7 to GPU 2 recorded in step 2). The first counterexample found in step 1 is along with bandwidth constraints (blocks 0 and 3 on link 1→4 and blocks 5 and 6 on link 4→1) along the output. The process iterates through verification, and if a counterexample is found, a search phase is initiated, converting the counterexample into constraints. These constraints are fed into the SMT solver to guide synthesis and checkpoint recovery, allowing the search to continue. Once a new candidate solution is found, the process iterates again. If no counterexample is found, the iteration stops, and the optimal solution is returned as the final synthesis result.

[0177] Step 6: Encode the scheduling results generated by the synthesizer into structured scheduling instructions and output them in XML format. Each scheduling instruction is... <step>The tags are used to represent the scheduling operation type, source and destination buffer information, data offset, data transfer volume, dependency identifier and other key fields, thereby ensuring the executability and semantic integrity of the scheduling logic.

[0178]

[0179] The meanings of each field are as follows:

[0180] s="0" indicates that the current scheduling step number is step 0;

[0181] The "type="rrc" indicates that this step is a "Remote Receive & Compute" operation, which is typically used to relay data so that the GPU can perform computation immediately after receiving the data.

[0182] srcbuf="o" and dstbuf="o" respectively indicate that both the source buffer and the destination buffer are output buffers.

[0183] srcoff="7" and dstoff="7" respectively indicate that the data is offset at the 7th cell in the source and destination buffers;

[0184] cnt="1" indicates that this operation involves the transmission of one data block unit;

[0185] depid="-1" and deps="-1" indicate that the current step is not bound to a specific preceding step number;

[0186] hasdep="1" indicates that the step has potential dependencies, but does not point to an explicit step number. It needs to be dynamically scheduled at runtime based on the arrival status of the data.

[0187] Step 7: Deployment of the Collective Communication Algorithm: During the deployment of the collective communication algorithm, it is necessary to map the block number on each GPU to its actual address offset on that GPU. For example, in this embodiment, the data block with block number 4 is initially stored on GPU4 and located at offset 0 within that device. To achieve this mapping relationship, the system configures the block number and actual offset in the algorithm in an XML file to ensure that the physical address location of each data block within the GPU can be accurately located during communication scheduling, ensuring the consistency and correctness of data blocks during scheduling and transmission.

[0188] During the deployment phase, by setting the environment variable NCCL_ALGO=MSCCL, the collective communication algorithm is specified as a custom format, enabling it to automatically recognize XML algorithm files.

[0189] In this example, the collective communication algorithm automatically synthesized based on the above steps was compared with the built-in algorithm of the mainstream NCCL communication library and the TACCL method, and its performance in terms of communication efficiency and synthesis time was comprehensively evaluated.

[0190] Figure 9 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement an efficient collective communication method for heterogeneous GPU clusters. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement an efficient collective communication method for heterogeneous GPU clusters. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0191] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.< / step>

Claims

1. An efficient collective communication method for a heterogeneous GPU cluster, the heterogeneous GPU cluster is configured with a plurality of heterogeneous GPU cluster nodes, each of which comprises a plurality of GPU nodes, characterized in that, The method comprises: acquiring a target communication operator set corresponding to the heterogeneous GPU cluster, and determining an initial state constraint and a target state constraint according to the target communication operator set; acquiring a physical topology corresponding to the heterogeneous GPU cluster, describing the physical topology according to a DSL rule to obtain a description result, and analyzing the description result through a built-in interpreter of the DSL rule to generate a directed graph model corresponding to the physical topology; the directed graph model is composed of a GPU node set and a physical topology interconnection relationship set; the GPU node set is composed of all GPU nodes in each heterogeneous GPU cluster node; the physical topology interconnection relationship set is composed of a first interconnection relationship and a second interconnection relationship; the first interconnection relationship is a relationship between GPU nodes within a heterogeneous GPU cluster node; the second interconnection relationship is a relationship between heterogeneous GPU cluster nodes; setting link bandwidth and link quantity on a plurality of the physical topology interconnection relationships; determining a minimum link bandwidth from a plurality of the link bandwidths; determining a maximum parallel data block number according to the minimum link bandwidth, the link bandwidth and the link quantity; when the GPU nodes are connected through shared links, the physical topology interconnection relationships formed by a plurality of shared links constitute a shared link group; determining a rated bandwidth of the shared link according to the link bandwidth and the link quantity corresponding to the shared link; selecting a communication scheme satisfying a constraint condition in the directed graph model by using a Z3 solver, the communication scheme being a transmission order of data on GPUs in the heterogeneous GPU cluster nodes; the communication scheme needs to satisfy the initial state constraint and the target state constraint of the target communication operator; determining a scheduling time cost of each communication scheme, and selecting a communication scheme corresponding to a minimum scheduling time cost in a plurality of the scheduling time costs as an efficient collective communication mode of the heterogeneous GPU cluster.

2. The efficient collective communication method for heterogeneous GPU cluster of claim 1, wherein, The physical topology contains parameters including a total number of heterogeneous GPU cluster nodes, a number of GPU nodes in the heterogeneous GPU cluster nodes, a first interconnection relationship between the GPU nodes in the heterogeneous GPU cluster nodes, and a second interconnection relationship between the heterogeneous GPU cluster nodes.

3. The efficient collective communication method for heterogeneous GPU cluster of claim 2, wherein, The parameters contained in the physical topology are specifically: n wherein, represents the total number of heterogeneous GPU cluster nodes, and the total number of heterogeneous GPU cluster nodes is specified as n when input. wherein, represents the number of GPUs within the i-th heterogeneous GPU cluster node, and is input to specify the number of GPUs contained in the i-th heterogeneous GPU cluster node as ; wherein, represents the first interconnection relationship between GPU nodes within the heterogeneous GPU cluster node i; represents the link type; the link type includes direct interconnection and switch interconnection ; represents the number of the source GPU; represents the number of the target GPU; represents the link bandwidth; the unit is GB / s; represents the number of links, which is a positive integer; wherein, represents the second interconnection relationship between the heterogeneous GPU cluster node i and the node j of the heterogeneous GPU cluster, and configures the link type between the heterogeneous GPU cluster nodes , is the GPU of the heterogeneous GPU cluster node i for external communication, is the GPU of the heterogeneous GPU cluster node j for external communication, represents the link bandwidth, and the unit is GB / s, represents the number of links, which is a positive integer.

4. The efficient collective communication method for heterogeneous GPU clusters according to claim 1, wherein, The initial state constraint and the target state constraint are as follows: Initial state constraint: Target state constraint: in, A collection of data blocks; Initial state constraints; For the g-th GPU; Constraints for the target state; For the first data block in the set indivual ; For the first data block in the set One data block.

5. The efficient collective communication method for heterogeneous GPU clusters according to claim 1, wherein, The constraint condition includes: Delivery rule constraint: the data block is sent from a source GPU to a target GPU, and the sending time should be earlier than the receiving time, that is, the time when the data block appears on the source GPU is less than the time when the data block appears on the target GPU; wherein, is a source GPU, is a target GPU, is a first of a set of data blocks, ; is a first of a set of data blocks, occurs on the source GPU, is a first of a set of data blocks, occurs on the target GPU; Direct link bandwidth constraint: more than the bandwidth limit cannot be sent on the same link at the same time; in, Indicates time The first in the inner data block set indivual Send from the source GPU to the target GPU; Shared link bandwidth constraint: the sum of a plurality of link bandwidths in each shared link group R cannot be greater than the rated bandwidth of the shared link group R; wherein, is a nominal bandwidth; is a data block in the set of data blocks; is a data block in the set of data blocks; .

6. The efficient collective communication method for heterogeneous GPU clusters according to claim 1, wherein, The minimum link bandwidth is implemented through the following expression: wherein, is the minimum link bandwidth; is the link bandwidth; is the set of physical topology interconnections, u and v are two GPU nodes or two heterogeneous GPU cluster nodes.

7. The efficient collective communication method for heterogeneous GPU cluster of claim 6, wherein, The maximum parallel data block number is implemented through the following expression: wherein, is the maximum number of parallel data blocks; is the link bandwidth; is the number of links; is the minimum link bandwidth.

8. The efficient collective communication method for heterogeneous GPU cluster of claim 6, wherein, The rated bandwidth is implemented through the following expression: wherein, is a nominal bandwidth of the shared link; is a link bandwidth of the shared link; is a number of links of the shared link; and are GPU nodes in the shared chain; is a shared link group.

9. The efficient collective communication method for heterogeneous GPU cluster of claim 6, wherein, The scheduling time cost of each communication scheme is determined through the following expression: wherein, is the time overhead of one scheduling of the data block in the communication scheme satisfying the constraint condition; represents the data amount of the data block, is the communication delay, is the inverse of the link bandwidth, is the data transmission time; represents the inverse of the computing throughput rate of the GPU, is the data processing time at the GPU end; represents the time cost; represents the total number of scheduling steps.

10. An efficient collective communication device for heterogeneous GPU clusters, comprising: It includes: The system input module is used to obtain a target communication operator set corresponding to the heterogeneous GPU cluster, and determine initial state constraints and target state constraints according to the target communication operator set; obtain a physical topology corresponding to the heterogeneous GPU cluster, describe the physical topology according to a DSL rule to obtain a description result, and analyze the description result through a built-in interpreter of the DSL rule to generate a directed graph model corresponding to the physical topology; the directed graph model is composed of a GPU node set and a physical topology interconnection relationship set; the GPU node set is composed of all GPU nodes in each heterogeneous GPU cluster node; the physical topology interconnection relationship set is composed of a first interconnection relationship and a second interconnection relationship, the first interconnection relationship is a relationship between GPU nodes in the heterogeneous GPU cluster node, and the second interconnection relationship is a relationship between the heterogeneous GPU cluster nodes; The comprehensive solver module is used to set link bandwidth and link quantity on a plurality of physical topology interconnection relationships; determine a minimum link bandwidth from a plurality of link bandwidths; determine a maximum parallel data block number according to the minimum link bandwidth, the link bandwidth and the link quantity; when the GPU nodes are connected through shared links, a plurality of shared links form a shared link group, and the physical topology interconnection relationship of the shared link group; determine a rated bandwidth of the shared link according to a link bandwidth and a link quantity corresponding to the shared link; select a communication scheme satisfying a constraint condition in the directed graph model by using a Z3 solver, the communication scheme is a transmission order of data on GPUs in the heterogeneous GPU cluster node; the communication scheme needs to satisfy the initial state constraints and the target state constraints of the target communication operator; The system output module is used to determine a scheduling time cost of each communication scheme, and select a communication scheme corresponding to a minimum scheduling time cost in a plurality of scheduling time costs as an efficient collective communication mode of the heterogeneous GPU cluster.

Images