Collective communication processing method and apparatus, computer device, and storage medium

Abstract

A collective communication processing method, comprising: acquiring a node orchestration request for a target collective operation in collective communication, wherein an original node list carried in the node orchestration request comprises server nodes selected for executing the target collective operation (202); acquiring topology information of the server nodes, wherein the topology information is used for describing switching nodes connected to the server nodes in hierarchies of a node communication network, and the server nodes communicate with each other on the basis of the connected switching nodes in the hierarchies (204); on the basis of the topology information, orchestrating and updating the arrangement order of the server nodes hierarchy by hierarchy according to the hierarchies in the node communication network so as to obtain an orchestrated node list (206); and according to the arrangement order of the server nodes in the orchestrated node list, performing collective communication for the target collective operation by means of the server nodes (208).

Description

Collective communication processing method, device, computer equipment and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2024102183123, filed on February 27, 2024, entitled “Collective Communication Processing Method, Apparatus, Computer Equipment and Storage Medium,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a collective communication processing method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0004] With the development of computer technology, distributed systems composed of multiple independent nodes can greatly improve computing efficiency and save overall computing time. For example, large-scale artificial intelligence models can be efficiently trained based on high-performance computing clusters composed of distributed servers.

[0005] In distributed systems, collective operations in collective communication enable efficient data exchange, state synchronization, and behavior coordination between nodes, enabling collaborative work and global computation. However, currently, when collective communication is performed between nodes in a distributed setup, the efficiency of collective operations between nodes is low. Summary of the Invention

[0006] According to various embodiments provided in the present application, a collective communication processing method, apparatus, computer device, computer-readable storage medium, and computer program product are provided.

[0007] In a first aspect, the present application provides a collective communication processing method, which is executed by a second server and includes:

[0008] Obtaining a node orchestration request for a target collective operation in collective communication, where the node orchestration request carries an original node list, and the original node list includes each server node selected to execute the target collective operation;

[0009] Obtaining topology information of each server node, where the topology information is used to describe the switch nodes connected to the server nodes at each level of the node communication network. The server nodes communicate with each other based on the switch nodes connected to them at each level of the node communication network.

[0010] Based on each topology information, the arrangement order of each server node is updated step by step according to the hierarchy in the node communication network to obtain an arrangement node list; and

[0011] According to the arrangement order of each server node in the orchestration node list, collective communication is performed through each server node for the target collective operation.

[0012] In a second aspect, the present application further provides a collective communication processing device, comprising:

[0013] A node orchestration request acquisition module is used to obtain a node orchestration request for a target collective operation in collective communication, wherein the node orchestration request carries an original node list, which includes each server node selected to execute the target collective operation;

[0014] A topology information acquisition module is used to obtain topology information of each server node. The topology information is used to describe the switch nodes connected to the server nodes at each level of the node communication network. The server nodes communicate with each other based on the switch nodes connected at each level of the node communication network.

[0015] A node arrangement update module is used to arrange and update the arrangement order of each server node step by step according to the hierarchy in the node communication network based on each topology information to obtain an arrangement node list; and

[0016] The collective operation processing module is used to perform collective communication for the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list.

[0017] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above collective communication processing method when executing the computer-readable instructions.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium having computer-readable instructions stored thereon, which implement the steps of the above collective communication processing method when executed by a processor.

[0019] In a fifth aspect, the present application also provides a computer program product, comprising computer-readable instructions, which implement the steps of the above collective communication processing method when executed by a processor.

[0020] In a sixth aspect, the present application provides a collective communication processing method, which is executed by a first server and includes:

[0021] When a target collective operation in collective communication is triggered, an original node list is determined, the original node list including each server node selected for performing the target collective operation;

[0022] Sending a node orchestration request generated according to the original node list to the server, wherein the node orchestration request is used to instruct the server to return an orchestration node list, where the orchestration node list is obtained by arranging and updating the arrangement order of each server node in the original node list; and

[0023] Receive the orchestration node list, and perform collective communication through each server node according to the target collective operation according to the arrangement order of each server node in the orchestration node list.

[0024] In a seventh aspect, the present application further provides a collective communication processing device, including:

[0025] The original node list acquisition module is used to determine the original node list when the target collective operation in the collective communication is triggered. The original node list includes each server node selected to perform the target collective operation;

[0026] A node orchestration request module is configured to send a node orchestration request generated based on the original node list to the server, wherein the node orchestration request is used to instruct the server to return an orchestration node list. The orchestration node list is based on the topology information of each server node and is updated step by step according to the hierarchy in the node communication network, with respect to the arrangement order of each server node in the original node list; and

[0027] The collective operation processing module is used to receive the orchestration node list and perform collective communication according to the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list.

[0028] In an eighth aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above collective communication processing method when executing the computer-readable instructions.

[0029] In a ninth aspect, the present application also provides a computer-readable storage medium having computer-readable instructions stored thereon, which implement the steps of the above collective communication processing method when executed by a processor.

[0030] In a tenth aspect, the present application also provides a computer program product, comprising computer-readable instructions, which implement the steps of the above collective communication processing method when executed by a processor.

[0031] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0033] FIG1 is a diagram illustrating an application environment of a collective communication processing method in some embodiments;

[0034] FIG2 is a schematic flow chart of a collective communication processing method in some embodiments;

[0035] FIG3A is a schematic diagram of a server node performing data transmission based on a switch node in some embodiments;

[0036] FIG3B is a schematic diagram illustrating changes in the arrangement and update of the original node list in some embodiments;

[0037] FIG4 is a schematic diagram of a flow chart of an orchestration update process in some embodiments;

[0038] FIG5 is a schematic flow chart of a collective communication processing method in some embodiments;

[0039] FIG6 is a diagram of a data center network topology in some embodiments;

[0040] FIG7 is a diagram of a data center network topology in some embodiments;

[0041] FIG8 is a logic flow loop diagram of the embodiment shown in FIG7;

[0042] FIG9 is a schematic diagram of network traffic in the embodiment shown in FIG7 ;

[0043] FIG10 is a logic flow loop diagram of the embodiment shown in FIG7 when not optimized;

[0044] FIG11 is a schematic diagram of network traffic when the embodiment shown in FIG7 is not optimized;

[0045] FIG12 is a diagram of a multi-tenant data center network topology in some embodiments;

[0046] FIG13 is a logic flow loop diagram of the embodiment shown in FIG12;

[0047] FIG14 is a schematic diagram of network traffic in the embodiment shown in FIG12 ;

[0048] FIG15 is a logic flow loop diagram of the embodiment shown in FIG12 when not optimized;

[0049] FIG16 is a schematic diagram of network traffic when the embodiment shown in FIG12 is not optimized;

[0050] FIG17 is a graph showing the relationship between the number of upstream flows on an access layer switch and the probability of congestion occurring in some embodiments;

[0051] FIG18 is a system architecture diagram of a collective communication processing method in some embodiments;

[0052] FIG19 is a flow chart of a collective communication processing method in some embodiments;

[0053] FIG20 is a schematic diagram of topological clustering in some embodiments;

[0054] FIG21 is a schematic diagram of topological clustering results in some embodiments;

[0055] FIG22 is a logic flow loop diagram of the embodiment shown in FIG21;

[0056] FIG23 is a graph comparing the net throughput of aggregate communication processing with and without optimization in some embodiments;

[0057] FIG24 is a block diagram of a collective communication processing device in some embodiments;

[0058] FIG25 is a block diagram of a collective communication processing device in some embodiments;

[0059] FIG26 is a diagram illustrating the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] The collective communication processing method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. The computer equipment in the application environment includes servers such as a first server 102 and a second server 104, and the first server 102 communicates with the second server 104 through a network. The data storage system can store the data that needs to be processed by each connected server. The data storage system can be set up separately, or integrated on each connected server, or placed on the cloud or other servers. When triggering a target collective operation in collective communication, the first server 102 can send a node orchestration request for the target collective operation to the second server 104. The node orchestration request carries an original node list, and the original node list includes each server node for the target collective operation, and the first server 102 is determined from each server node. After receiving the node orchestration request, the second server 104 orchestrates and updates the arrangement order of each server node for the target set operation in the original node list carried by the node orchestration request, based on the topology information of each server node and according to the hierarchy in the node communication network, to obtain an orchestration node list. The second server 104 can return the orchestration node list to the first server 102, so that the first server 102 can perform collective communication for the target set operation through each server node according to the arrangement order of each server node in the orchestration node list.

[0062] In addition, when a target collective operation in collective communication is triggered, first server 102 determines an original node list including each server node for the target collective operation. First server 102 sends a node orchestration request generated based on the original node list to second server 104, and receives an orchestration node list returned by second server 104, which is obtained by orchestrating and updating the arrangement order of each server node in the original node list. First server 102 then performs collective communication for the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list. In some embodiments, the collective communication processing method can also be implemented solely by first server 102, that is, first server 102 directly performs orchestration and update processing on the arrangement order of each server node, and performs collective communication based on the obtained orchestration node list.

[0063] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In addition, at least one of the first server 102 and the second server 104 in this application environment can be replaced by a terminal. The terminal can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, etc.

[0064] In an exemplary embodiment, as shown in FIG2 , a collective communication processing method is provided. The method is executed by a computer device, specifically, a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to the second server in FIG1 as an example for description, and includes the following steps 202 to 208. Among them:

[0065] Step 202: Obtain a node orchestration request for a target collective operation in collective communication. The node orchestration request carries an original node list. The original node list includes each server node selected to execute the target collective operation.

[0066] Collective communication is a global communication operation involving all processes in a process group and capable of handling a large amount of inter-process communication, synchronization, and computation. Collective communication is particularly critical in distributed systems, as extensive communication is often required between nodes. For example, in the distributed training of deep learning, collective communication can be used to transmit large amounts of data, such as network model weight parameters and temporary variables generated during training, to improve model training efficiency. Basic collective communication operations include Send, Receive, Copy, intra-group process barrier synchronization, and inter-node process synchronization (Signal+Wait). These basic operations can be combined to form various collective operations. Collective operations are global communication operations performed collaboratively by a group of processes, allowing all processes in the process group to participate in communication and jointly operate on data. Collective operations may include, but are not limited to, at least one of various communication operations, including broadcast, gather, many-to-many gather (allgather), scatter, reduce, many-to-many reduce (allreduce), reduce-scatter, and many-to-many (all-to-all). The target collective operation refers to the collective operation that needs to be executed in the currently triggered collective communication, such as a many-to-many reduce operation.

[0067] A node orchestration request is used to request an orchestration update for the order of server nodes for a target set operation. Server nodes are the nodes selected to implement the target set operation. This means that the target set operation is performed between selected server nodes, such as those that can perform many-to-many reduction operations. The node orchestration request carries an original node list, which includes the server nodes selected to perform the target set operation.

[0068] Specifically, the second server can obtain a node orchestration request for a target set operation in a set communication, and specifically can receive the node orchestration request sent by the first server through a network. The first server can be a node among the various server nodes for the target set operation. In some embodiments, the first server can be a node among the various server nodes for performing the target set operation. The second server acts as a server for orchestrating and updating the various server nodes for the target set operation. The second server can parse the node orchestration request to obtain an original node list. The original node list includes the various server nodes for the target set operation, that is, the original node list includes the various server nodes selected for performing the target set operation. In specific applications, the original node list can record the node identification of each server node, such as at least one of the identification information of the server node, such as the node name, node number, etc.

[0069] Step 204: Acquire topology information of each server node. The topology information is used to describe the switch nodes connected to the server nodes at each level of the node communication network. The server nodes communicate with each other based on the switch nodes connected at each level of the node communication network.

[0070] The topology information describes the switching nodes to which the server nodes are connected at each level of the node communication network, so that communication at the corresponding level can be carried out through the connected switching nodes. The server nodes can be directly connected to the switching nodes for communication, or indirectly connected through switching nodes in other levels. For example, the topology information can include the node identifiers of the switching nodes to which the server nodes are connected at each level of the node communication network. The node communication network can include multiple switching nodes for forwarding data packets between servers, and the switching nodes can be implemented based on at least one of the following devices: terminals, servers, switches, etc. The switching nodes in the node communication network can be constructed based on a multi-level structure, with each level including at least one switching node, and the server nodes communicate with each other based on the switching nodes connected to each level. The node communication network can record the data routing method used when communicating between the server nodes, that is, data is transmitted through the corresponding switching nodes in the node communication network to achieve communication between different server nodes.

[0071] Exemplarily, for each server node in the original node list, the second server can obtain topology information for each server node in the node communication network. For example, the second server can query and determine the topology information for each server node in the node communication network based on the node identifier of each server node. The node communication network is used to support communication between the server nodes and can include multiple levels. Each level can include switching nodes for communication, thereby enabling communication between the server nodes through the switching nodes in the multiple levels. The topology information can record the switching nodes to which the server nodes are connected in each level. For example, the topology information can be {H0:1,1,1}, which describes that server node H0 is connected to switching node L1 in the first level of the node communication network, switching node P1 in the second level, and switching node C1 in the third level. That is, server node H0 can communicate with other server nodes in the node communication network through switching node L1 in the first level, switching node P1 in the second level, and switching node C1 in the third level. In some embodiments, high-level switching nodes are used to implement cross-node communication between low-level switching nodes and downstream server nodes. For example, when implementing communication between the downstream server node of switching node P1 in the second level and the downstream server node of switching node P2 in the second level across switching nodes in the second level, it is necessary to implement it based on the switching nodes in the third level, such as it can be implemented based on the switching node C1 in the third level.

[0072] In a specific implementation, as shown in Figure 3A, the node communication network may include two layers: the first layer is the L layer, and the second layer is the S layer. The L layer includes switching nodes L0 and L1. Switching node L0 is connected to server nodes H0, H1, H2, and H3, while switching node L1 is connected to server nodes H4, H5, H6, and H7. The S layer may also be divided into switching nodes S0 and S1 to support cross-node communication between the server nodes connected to switching nodes L0 and L1, respectively. In other words, server nodes H0, H1, H2, and H3 can communicate directly with switching node L0 and indirectly with switching node S0 through switching node L0. As shown in 301 in FIG3A , when communication occurs between server node H0 and server node H1, both server node H0 and server node H1 are connected to switching node L0 in the L layer. Therefore, communication between server node H0 and server node H1 can be directly implemented based on switching node L0 in the L layer, i.e., data packets between server node H0 and server node H1 can be forwarded via switching node L0. However, when communication occurs between server node H0 and server node H7, as shown in 302 in FIG3A , the two need to cross different switching nodes in the L layer, so cross-node communication is required via switching nodes in the S layer. Specifically, data packets between server node H0 and server node H7 can be forwarded via switching node L0 in the L layer, switching node S0 in the S layer, and switching node L1 in the L layer. This allows cross-node communication between different switching nodes in the L layer, each connected to a server node, to be implemented via the switching nodes in the S layer. Compared with the communication between server node H0 and server node H1 through switching node L0, the communication between server node H0 and server node H7 needs to be implemented through the S-layer switching node across switching nodes L0 and L1. The number of switching nodes involved in the communication process increases, which increases the communication delay.

[0073] Step 206 , based on each topology information, the arrangement order of each server node is updated step by step according to the hierarchy in the node communication network to obtain an arrangement node list.

[0074] The arrangement node list is updated based on the original node list, and can be specifically re-arranged based on the arrangement order of each server node. When the arrangement order of each server node is updated step by step, it can be implemented according to the hierarchical dimension from high to low or from low to high. For example, when the node communication network includes N levels, the second server can arrange and update the arrangement order of each server node step by step according to the hierarchical dimension from the first level to the Nth level, or it can arrange and update the arrangement order of each server node step by step according to the hierarchical dimension from the Nth level to the first level.

[0075] Optionally, the second server can perform orchestration and update of the arrangement order of each server node, specifically based on the respective topological information of each server, and perform orchestration and update step by step according to the hierarchy in the node communication network to rearrange the arrangement order of each server node and obtain an orchestration node list. The arrangement order of each server node in the orchestration node list can be determined based on the hierarchical dimension of the step-by-step orchestration update. As shown in FIG3B , in the original node list carried by the node orchestration request, there are eight server nodes including H0, H1, H2, H3, H4, H5, H6 and H7. The respective topological information of the eight server nodes is shown in FIG3A . In the original node list, the arrangement order of each server node is H4-H5-H3-H0-H6-H2-H7-H1; the second server can perform orchestration and update of the arrangement order of each server node according to the respective topological information of each server node, such as performing orchestration and update of the arrangement order of each server node step by step in the order from the S layer to the L layer. Specifically, for the S layer, the arrangement can be performed in the order of S0-S1, that is, the server nodes {H3, H0, H2, H1} belonging to the switching node S0 are arranged first, and then the server nodes {H4, H5, H6, H7} belonging to the switching node S1 are arranged; after the S layer arrangement is completed, the arrangement is performed in the order of L0-L1 on the L layer, that is, the server nodes belonging to the switching node L0 are arranged first, and then the server nodes belonging to the switching node L1 are arranged, so that the orchestration node list can be obtained. The arrangement order of each server node in the orchestration node list is H0-H1-H2-H3-H4-H5-H6-H7. By arranging and updating the arrangement order of each server node step by step according to the hierarchy in the node communication network, it can be ensured that each server node in the arrangement node list is arranged step by step according to the hierarchy in the node communication network. Therefore, when collective communication is performed for the target collective operation through each server node according to the arrangement order of each server node, communication between nodes can be reduced, and the communication delay of the collective operation in the collective communication can be reduced, thereby improving the efficiency of the collective operation.

[0076] Step 208 : performing collective communication for the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list.

[0077] Exemplarily, for the orchestration node list, the arrangement order of each server node is re-arranged and updated, and the second server performs collective communication for the target set operation through each server node according to the arrangement order of each server node. For example, each server node can perform the target set operation in turn according to the arrangement order of each server node in the orchestration node list, such as performing many-to-many protocol operations in turn.

[0078] In the above-mentioned collective communication processing method, for each server node in the original node list carried by the node orchestration request for the target collective operation, the arrangement order of each server node is orchestrated and updated step by step based on the topological information of each server node and the hierarchy in the node communication network to obtain an orchestration node list, and collective communication is performed for the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list. By orchestrating and updating the arrangement order of each server node in the original node list step by step according to the hierarchy in the node communication network, the arrangement order can be adjusted according to the topological information of each server node, and collective communication can be performed for the target collective operation based on the arrangement order of each server node after the orchestration update. This can reduce the generation of cross-node communication, reduce the communication delay of the collective operation in the collective communication, and thus improve the efficiency of the collective operation.

[0079] In an exemplary embodiment, as shown in FIG4 , the orchestration update process is to orchestrate and update the arrangement order of each server node step by step according to the hierarchy in the node communication network based on each topology information to obtain an orchestration node list, including:

[0080] Step 402 : Based on each topology information, the server nodes are hierarchically clustered step by step according to the levels in the node communication network to obtain a topology clustering result of each server node.

[0081] The switching nodes in the node communication network are divided into multiple levels. When the server nodes connected to the switching nodes of the same level communicate, they need to communicate through the switching nodes of the previous level. By hierarchically clustering the server nodes according to the level of each switching node in the node communication network, specifically in a hierarchical order from low to high or from high to low, the topological structure relationship of each server node in the node communication network can be accurately determined. For example, the switching nodes required for communication between the server nodes can be determined. The topological clustering results can include the hierarchical clustering results of the server nodes at each level.

[0082] Exemplarily, the second server can hierarchically cluster each server node according to the level in the node communication network. For example, when the node communication network includes two levels, the second server can hierarchically cluster each server node according to the order from the first level to the second level based on the topological information of each server node, and obtain a topological clustering result of each server node. In a specific implementation, the second server can hierarchically cluster each server node according to the first level based on the topological information of each server node, and divide the server nodes connected to the same interactive node in the first level into the same class, and obtain multiple clustering results of the first level. For each clustering result of the first level, the second server can hierarchically cluster each server node according to the second level based on the topological information of each server node, and divide the server nodes connected to the same interactive node in the second level into the same class, and obtain multiple clustering results of the second level. After each first-level clustering result is hierarchically clustered according to the second level, multiple second-level clustering results can be obtained. After traversing each first-level clustering result, the second-level clustering results of each first-level clustering result can be obtained. The second server can obtain the topological clustering results of each server node based on the second-level clustering results of each first-level clustering result.

[0083] Step 404 : Based on the topological clustering results, node search is performed level by level according to the hierarchy in the node communication network to obtain an orchestration node list. The arrangement order of the server nodes in the orchestration node list is the same as the hierarchical order of the node search.

[0084] The node search can be performed by searching the nodes in the node communication network in a descending order or in a descending order. The arrangement order of the server nodes in the orchestration node list is the same as the hierarchical order of the node search, that is, the arrangement order of the server nodes can be determined based on the hierarchical order of the node search. For example, in the orchestration node list, the server nodes can be arranged in sequence according to the hierarchical order of the node search.

[0085] Optionally, the second server can further perform a node search based on the topological clustering results of each server node, specifically performing a node search step by step according to the levels in the node communication network. In a specific implementation, the hierarchical order of the node search can be the same as or different from the hierarchical order of the hierarchical clustering. For example, hierarchical clustering can be implemented step by step according to the hierarchical order from high to low in the node communication network, while the node search can be implemented step by step according to the hierarchical order from low to high in the node communication network. For example, when the node communication network includes two levels, the second server can perform a node search for each server node step by step in the order from the second level to the first level based on the topological clustering results to obtain an orchestration node list. In a specific implementation, the second server can perform a node search according to the second level based on the results of each topological clustering, that is, search for a topological cluster a1 connected to a switching node A1 in the second level and a topological cluster a2 connected to a switching node A2 in the second level based on the results of each topological clustering, each server node in the topological cluster a1 is connected to the switching node A1 in the second level, and each server node in the topological cluster a2 is connected to the switching node A2 in the second level; the second server performs a node search according to the first level in the topological cluster a1 and the topological cluster a2 respectively, that is, searches for a topological cluster b1 connected to the switching node B1 in the first level and a topological cluster b2 connected to the switching node B2 in the first level in the topological cluster a1, and searches for a topological cluster b3 connected to the switching node B3 in the first level and a topological cluster b4 connected to the switching node B4 in the first level in the topological cluster a2, so that {a1[(b1), (b2)], a2[(b3), (b4)]} can be obtained by searching. Among them, (b1) and (b2) are obtained by further searching nodes at the first level based on a1, and (b3) and (b4) are obtained by further searching nodes at the first level based on a2. As a result, the arrangement node list is {(b1), (b2), (b3), (b4)}, that is, arranged in the order of (b1)-(b2)-(b3)-(b4).

[0086] In some embodiments, different node searches can result in different orchestration node lists. For example, after searching for topological cluster a1 connected to a switching node A1 in the second level and topological cluster a2 connected to a switching node A2 in the second level based on the results of each topological cluster, a further node search can be performed on topological cluster a2 to obtain topological cluster b3 connected to switching node B3 in the first level and topological cluster b4 connected to switching node B4 in the first level. Further node searches can then be performed on topological cluster a1 to obtain an orchestration node list of {(b3), (b4), (b1), (b2)}, i.e., arranged in the order of (b3)-(b4)-(b1)-(b2). In a specific application, when (b1), (b2), (b3), and (b4) include multiple server nodes, the multiple server nodes in (b1), (b2), (b3), and (b4) can be arbitrarily sorted. For example, when b1 includes three server nodes, b11, b12, and b13, in the orchestration node list, the inside of (b1) can be (b11, b12, b13), (b11, b13, b12), (b12, b11, b13), (b13, b11, b12), (b12, b13, b11), or (b13, b12, b11).

[0087] In this embodiment, the second server hierarchically clusters each server node according to the hierarchy in the node communication network based on the topological information of each server node, and performs node search step by step based on the topological clustering results obtained by hierarchical clustering, so as to obtain an orchestration node list in which the arrangement order of each server node conforms to the hierarchical order of the node search. This ensures that each server node in the orchestration node list is arranged step by step according to the hierarchy in the node communication network, so that when collective communication is performed for the target collective operation through each server node according to the arrangement order of each server node, communication between nodes can be reduced, and the communication delay of the collective operation in the collective communication can be reduced, thereby improving the efficiency of the collective operation.

[0088] In an exemplary embodiment, based on each topology information, each server node is hierarchically clustered step by step according to the hierarchy in the node communication network to obtain a topological clustering result of each server node, including: based on each topology information, each server node is hierarchically clustered step by step according to the hierarchy dimension from high to low in the node communication network to obtain a topological clustering result of each server node; in the node communication network, high-level switching nodes are used to support cross-node communication between server nodes connected to low-level switching nodes.

[0089] The node communication network is configured with multiple hierarchies, each including at least one switching node. Higher-level switching nodes support cross-node communication between server nodes connected to lower-level switching nodes. Specifically, within the same hierarchical level, when server nodes connected to different switching nodes communicate with each other, cross-node communication must occur across the switching nodes of that level, specifically through switching nodes at a higher level. For example, within the same hierarchical level, switching node A connects to server node 1, and switching node B connects to server node 2. When communication occurs between server nodes 1 and 2, because server nodes 1 and 2 are connected to different switching nodes within that level, communication must occur across the switching nodes within that level. This communication can be accomplished through switching node C in the next higher level. The specific data transmission route can then be server node 1 - switching node A - switching node C - switching node B - server node 2. Hierarchical clustering can be implemented from high to low hierarchical dimensions or from low to high hierarchical dimensions.

[0090] Exemplarily, the second server can determine the hierarchical order of hierarchical clustering, specifically, based on the topological information of the server nodes, hierarchically cluster each server node step by step according to the hierarchical dimension from high to low, to obtain the topological clustering results of each server node. Each topological clustering result can include the clustering results of the hierarchical clustering of the server nodes at each level. For example, when the node communication network includes three levels, hierarchical clustering can be performed step by step according to the hierarchical dimension from the first level to the second level to the third level, or hierarchical clustering can be performed step by step according to the hierarchical dimension from the third level to the second level to the first level.

[0091] In this embodiment, the second server hierarchically clusters each server node according to the hierarchical dimension from high to low, which can accurately determine the clustering results of each server node in each level, which is conducive to accurately arranging each server node step by step according to the level in the node communication network, thereby reducing communication across nodes and reducing the communication delay of collective operations in collective communications, thereby improving the efficiency of collective operations.

[0092] In an exemplary embodiment, the levels in the node communication network include a core layer, a convergence layer, and an access layer from high to low; based on each topology information, each server node is hierarchically clustered step by step according to the level dimensions from high to low in the node communication network to obtain a topological clustering result of each server node, including: clustering each server node according to the level dimension of the core layer based on each topology information to obtain a core layer clustering result of each server node; clustering each server node according to the level dimension of the convergence layer based on the core layer clustering result to obtain a convergence layer clustering result of each server node; clustering each server node according to the level dimension of the access layer based on the convergence layer clustering result to obtain a topological clustering result of each server node.

[0093] The layers in the node communication network include, from high to low, the core layer (Core), the aggregation layer (Spine), and the access layer (Leaf). The core layer can be the third layer, the aggregation layer is the second layer, and the access layer is the first layer. That is, the switching nodes in the core layer are used to support cross-node communication between server nodes connected to the switching nodes in the second layer or the first layer; the switching nodes in the aggregation layer are used to support cross-node communication between server nodes connected to the switching nodes in the first layer. When performing hierarchical clustering step by step, it can be implemented in the order of the core layer, the aggregation layer, and the access layer.

[0094] Exemplarily, for a node communication network including a core layer, a convergence layer, and an access layer from high to low, the second server can cluster each server node according to the hierarchical dimension of the core layer based on each topology information, specifically dividing the server nodes connected to the same switching node in the core layer into the same cluster, and obtaining a core layer clustering result for each server node. The second server clusters each server node according to the hierarchical dimension of the convergence layer based on each core layer clustering result, specifically dividing the server nodes connected to the same switching node in the convergence layer into the same cluster, and obtaining a convergence layer clustering result. The second server clusters each server node according to the hierarchical dimension of the access layer based on each convergence layer clustering result, specifically dividing the server nodes connected to the same switching node in the access layer into the same cluster, and obtaining a topological clustering result for each server node.

[0095] In this embodiment, for a node communication network including a core layer, a convergence layer and an access layer from high to low, the second server clusters each server node in sequence according to the hierarchical order of the core layer, the convergence layer and the access layer, and obtains a topological clustering result that reflects the clustering results in each layer, which is conducive to accurately arranging each server node step by step according to the hierarchy in the node communication network, thereby reducing communication across nodes, reducing the communication delay of collective operations in collective communications, and thus improving the efficiency of collective operations.

[0096] In an exemplary embodiment, based on each topological clustering result, a node search is performed level by level according to the hierarchy in the node communication network to obtain an orchestration node list, including: based on each topological clustering result, a node search is performed level by level for each server node according to the hierarchy dimension from high to low in the node communication network to obtain an orchestration node list; in the node communication network, high-level switching nodes are used to support cross-node communication between server nodes connected to low-level switching nodes.

[0097] The node communication network consists of multiple layers, each containing at least one switching node. Switching nodes at higher layers support inter-node communication between server nodes connected to switching nodes at lower layers. Specifically, within a layer, when server nodes connected to different switching nodes communicate with each other, this communication must bypass switching nodes within that layer. This communication can be accomplished through switching nodes at higher layers.

[0098] Specifically, the second server can determine the hierarchical order of the node search. Specifically, based on the topological clustering results of each server node, the second server can search each server node level by level according to the hierarchical dimension from high to low, thereby obtaining an orchestration node list for each server node. For example, when the node communication network includes three levels, the node search can be performed level by level based on the topological clustering results from the first level to the second level to the third level, or the node search can be performed level by level based on the hierarchical dimension from the third level to the second level to the first level.

[0099] In this embodiment, the second server searches for each server node according to the hierarchical dimension from high to low, and accurately arranges each server node step by step according to the hierarchy in the node communication network, thereby reducing communication across nodes and reducing the communication delay of collective operations in collective communications, thereby improving the efficiency of collective operations.

[0100] In an exemplary embodiment, the levels in the node communication network include, from high to low, a core layer, a convergence layer, and an access layer; based on the results of each topological clustering, according to the hierarchical dimensions from high to low in the node communication network, a node search is performed for each server node step by step to obtain an orchestration node list, including: searching according to the hierarchical dimension of the core layer based on the results of each topological clustering to obtain a core layer search result; searching according to the hierarchical dimension of the convergence layer based on the core layer search result to obtain a convergence layer search result; searching according to the hierarchical dimension of the access layer based on the convergence layer search result to obtain an access layer search result; obtaining an orchestration node list based on the core layer search result, the convergence layer search result, and the access layer search result.

[0101] Among them, the levels in the node communication network include, from high to low, the core layer, the convergence layer, and the access layer. Optionally, for a node communication network including, from high to low, the core layer, the convergence layer, and the access layer, the second server may search according to the hierarchical dimension of the core layer based on each topological clustering result, specifically for each switching node in the core layer, search for the server nodes connected to the corresponding switching node based on each topological clustering result, and obtain the core layer search results. Among them, the core layer search results may record the server nodes connected to the switching nodes in the core layer; the server nodes connected to the switching nodes may be direct communication connections or indirect communication connections through switching nodes in other levels. The second server searches according to the hierarchical dimension of the convergence layer based on the core layer search results, specifically for each switching node in the convergence layer, search for the server nodes connected to the corresponding switching node based on each core layer search result, and obtain the convergence layer search results. The convergence layer search results may include server nodes connected to the switching nodes in the convergence layer. The second server searches according to the hierarchical dimension of the access layer based on the convergence layer search results. Specifically, for each switching node in the access layer, the second server searches for the server nodes connected to the corresponding switching node based on each convergence layer search result to obtain the access layer search results. The access layer search results may include server nodes connected to the switching nodes in the access layer.

[0102] The second server can obtain an orchestration node list by combining the core layer search results, the convergence layer search results, and the access layer search results. Specifically, the second server can sort and combine the server nodes according to the hierarchical search order of the core layer, the convergence layer, and the access layer, and obtain an orchestration node list. For example, the core layer search results include c1, c2, and c3, which correspond to the switching nodes C1, C2, and C3 in the core layer, respectively; for c1, c2, and c3, in the convergence layer search results obtained after searching according to the hierarchical dimension of the convergence layer, s1 and s2 are obtained based on the c1 search, s3 and s4 are obtained based on the c2 search, and s5 and s6 are obtained based on the c3 search. S1, s2, s3, s4, s5, and s6 correspond to the switching nodes S1, S2, S3, S4, S5, and S6 in the convergence layer, respectively; for s1, s2, s3, s4, s5, and s6, which correspond to the access layer switching nodes S1, S2, S3, S4, S5, and S6, respectively. After searching the hierarchical dimension of the layer, the access layer search results are obtained. Based on s1, l1 and l2 are searched, based on s2, l3 and l4 are searched, based on s3, l5 and l6 are searched, based on s4, l7 and l8 are searched, based on s5, l9 and l10 are searched, and based on s6, l11 and l12 are searched. l1, l2, l3, l4, l5, l6, l7, l8, l9, l10, l11 and l12 correspond to the switching nodes L1, L2, L3, L4, L5, L6, L7, L8, L9, L10, L11 and L12 in the access layer, respectively. The second server can sort the server nodes according to c1, c2, and c3, s1, s2, s3, s4, s5, and s6, and l12, l1, l2, l3, l4, l5, l6, l7, l8, l9, l10, l11, and l12, to obtain an orchestration node list in which the order of sorting the server nodes conforms to the hierarchical order of the node search. For example, the orchestration node list can be {l1, l2, l3, l4, l5, l6, l7, l8, l9, l10, l11, l12}, or can be {l12, l11, l10, l9, l8, l7, l6, l5, l4, l3, l2, l1}.

[0103] In this embodiment, for a node communication network including a core layer, a convergence layer, and an access layer from high to low, the second server searches for the topological clustering results in sequence according to the hierarchical order of the core layer, the convergence layer, and the access layer to obtain an orchestration node list, and can arrange each server node step by step according to the hierarchy in the node communication network, thereby reducing communication across nodes, reducing the communication delay of collective operations in collective communications, and thus improving the efficiency of collective operations.

[0104] In an exemplary embodiment, obtaining a node orchestration request for a target collective operation in collective communication includes: determining a collective communication library in the collective communication; and obtaining a node orchestration request for the target collective operation in the collective communication based on a communication interface with the collective communication library.

[0105] Among them, the collective communication library is a library for supporting communication between multiple processes in parallel computing. In a parallel computing environment, multiple processes may need to work together to complete a task, and the collective communication library provides efficient collective operations so that these processes can easily perform data exchange, synchronization and other operations. Exemplarily, the second server can determine the collective communication library in the collective communication, which includes various collective operations, that is, implement the corresponding collective operations based on the collective communication library. The collective communication library may include but is not limited to at least one of the communication libraries including MPI (Message Passing Interface), NCCL (NVIDIA Collective Communications Library), Libfabric (Asynchronous Communication Library) or TBB (Intel Threading Building Blocks). The second server obtains a node orchestration request for the target collective operation through the communication interface between the second server and the collective communication library, and the second server can be implemented based on a plug-in of the collective communication library through the communication interface between the second server and the collective communication library.

[0106] In this embodiment, the second server obtains the node orchestration request for the target set operation through the communication interface with the set communication library, so that the node orchestration request including the original node list can be obtained accurately and quickly, which is conducive to improving the efficiency of the set communication for the target set operation.

[0107] In an exemplary embodiment, obtaining topology information of each server node includes: determining a topology database associated with a node communication network; and querying the topology information of each server node from the topology database according to the original node list.

[0108] The topology database is used to record the hierarchical topology structure in the node communication network and the server nodes to which each switching node is connected. Specifically, the second server can determine the topology database associated with the node communication network and, based on the original node list carried in the node orchestration request, query the topology database to obtain the topology information of each server node. In a specific implementation, the original node list can include the node identifiers of each server node operated on the target set. The second server can query the topology database based on the node identifiers to obtain the topology information of the identified server nodes. The topology information can include the node identifiers of the server nodes and the node identifiers of the connected switching nodes.

[0109] In this embodiment, the second server obtains the topology information of each server node from the topology database associated with the node communication network according to the original node list, so that the topology information of each server node can be used to arrange and update the arrangement order of each server node step by step according to the hierarchy in the node communication network, and perform collective communication for the target collective operation based on the arrangement order of each server node after the arrangement update, which can reduce the generation of cross-node communication and reduce the communication delay of collective operation in collective communication, thereby improving the efficiency of collective operation.

[0110] In an exemplary embodiment, obtaining a node orchestration request for a target collective operation in collective communication includes obtaining the node orchestration request from a target server node, where the target server node is determined from server nodes selected for performing the target collective operation in collective communication.

[0111] The target server node is determined from among the server nodes performing the target collective operation, and specifically may include the server node that initiates collective communication for the target collective operation. Exemplarily, the second server determines the target server node and obtains a node orchestration request from the target server node. In a specific implementation, the node orchestration request may be proactively sent by the target server node to the second server, and the second server may directly receive the node orchestration request sent by the target server node.

[0112] Furthermore, according to the arrangement order of each server node in the orchestration node list, collective communication is performed through each server node for the target collective operation, including: sending the orchestration node list to the target server node, the orchestration node list is used to instruct the target server node to perform collective communication according to the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list.

[0113] Specifically, after orchestrating and updating the original node list carried in the node orchestration request sent by the target server node, the second server can send the orchestrated node list to the target server node, so that the target server node can perform collective communication according to the target collective operation through each server node in the arrangement order of the server nodes in the orchestration node list, that is, the target server node initiates collective communication processing for the target collective operation.

[0114] In this embodiment, the second server obtains a node orchestration request from the target server node and sends the orchestration node list to the target server node, so that the target server node initiates collective communication for the target collective operation based on the orchestration node list. This can reduce cross-node communication and reduce the communication delay of collective operations in collective communication, thereby improving the efficiency of collective operations.

[0115] In an exemplary embodiment, the collective communication processing method further includes: when each server node for the target collective operation meets the orchestration update trigger condition, executing the step of obtaining a node orchestration request for the target collective operation in the collective communication.

[0116] Among them, the orchestration update trigger condition is used to determine whether it is necessary to trigger the orchestration update processing for the server node. For example, when the server node for the target set operation is updated, it is considered that the orchestration update trigger condition is met. For example, when there is a faulty node in the server nodes for the target set operation, or when a new server node is added, it can be considered that the orchestration update trigger condition is met. Exemplarily, the second server can monitor the various server nodes for the target set operation, and specifically detect the state changes of each server node. When it is detected that the orchestration update trigger condition is met, such as when the second server detects that there is a faulty node in the server nodes for the target set operation, or when a new server node is added, the server executes the step of obtaining the node orchestration request for the target set operation in the set communication, and timely orchestrates the arrangement order of the server nodes whose states have changed.

[0117] In this embodiment, when it is detected that the orchestration update trigger condition is met, the second server executes the step of obtaining a node orchestration request for the target collective operation in the collective communication, and can timely orchestrate and update the arrangement order of each server node whose status has changed, thereby helping to improve the efficiency of the collective operation.

[0118] In an exemplary embodiment, as shown in FIG5 , a collective communication processing method is provided. The method is executed by a computer device, specifically, a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to the first server in FIG1 as an example for description, and includes the following steps 502 to 506. Among them:

[0119] Step 502: When a target collective operation in collective communication is triggered, an original node list is determined. The original node list includes each server node selected for executing the target collective operation.

[0120] The target collective operation refers to the collective operation currently triggered in the collective communication that needs to be executed. The original node list includes the server nodes selected to implement the target collective operation. The server nodes are the nodes selected to implement the target collective operation. That is, the target collective operation is implemented between the selected server nodes.

[0121] For example, when a target collective operation in collective communication is triggered, indicating that the target collective operation needs to be performed, the first server may determine a corresponding original node list, where the original node list includes server nodes for the target collective operation, and the first server is a node among the server nodes for the target collective operation.

[0122] In step 504, a node orchestration request generated based on the original node list is sent to the server. The node orchestration request is used to instruct the server to return an orchestration node list. The orchestration node list is based on the topology information of each server node and is updated step by step according to the hierarchy in the node communication network, with respect to the arrangement order of each server node in the original node list.

[0123] The node orchestration request is generated based on the original node list and is used to request an orchestration update of the arrangement order of each server node for target set operations. Topology information describes the switch nodes to which the server nodes are connected at each level of the node communication network, so that communication at the corresponding level can be carried out through the connected switch nodes. The node communication network may include multiple switch nodes for forwarding data packets between servers. The switch nodes in the node communication network may be constructed based on a multi-level structure, with each level including at least one switch node, and the server nodes communicate with each other based on the switch nodes connected at each level. When the arrangement order of each server node is updated step by step, it can be implemented in a hierarchical dimension from high to low or from low to high.

[0124] Optionally, the first server generates a node orchestration request based on the original node list and sends the node orchestration request to the server to instruct the server to orchestrate and update the arrangement order of each server node in the original node list step by step according to the hierarchy in the node communication network based on the topology information of each server node, and return the orchestration node list obtained by the orchestration update.

[0125] Step 506: Receive the orchestration node list, and perform collective communication according to the target collective operation through each server node in the arrangement order of each server node in the orchestration node list.

[0126] Specifically, for the orchestration node list, the arrangement order of each server node is re-arranged and updated, and the first server performs collective communication for the target set operation through each server node according to the arrangement order of each server node. For example, the first server can initiate the target set operation according to the arrangement order of each server node in the orchestration node list to control each server node to perform the target set operation in sequence, such as performing many-to-many protocol operations in sequence.

[0127] In the above-described collective communication processing method, when a target collective operation in collective communication is triggered, an original node list including each server node for the target collective operation is determined, a node orchestration request generated based on the original node list is sent to the server, and an orchestration node list returned by the server is received, which is obtained by orchestrating and updating the arrangement order of each server node in the original node list. Collective communication is then performed for the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list. For each server node for the target collective operation, an orchestration node list is obtained by orchestrating and updating the arrangement order of each server node step by step according to the hierarchy in the node communication network, and collective communication is then performed for the target collective operation according to the arrangement order of each server node in the orchestration node list. This can reduce cross-node communication, reduce the communication latency of collective operations in collective communication, and thus improve the efficiency of collective operations.

[0128] This application also provides an application scenario, which applies the above-mentioned collective communication processing method. Specifically, the application of the collective communication processing method in this application scenario is as follows:

[0129] Modern data center networks generally adopt a Clos architecture, which employs three or two layers of switching nodes, specifically switches. As shown in Figure 6, this architecture includes three layers of switches: the access layer, the aggregation layer, and the core layer, from the lowest to the highest. Furthermore, one layer of server nodes, specifically servers (hosts), is used. The core layer includes eight switches (i.e., switching nodes) from C0 to C7, the aggregation layer includes 16 switches from S0 to S15, the access layer includes 16 switches from L0 to L15, and the server layer includes 32 servers from H0 to H31. Switches are solely responsible for forwarding data packets, while all business applications are deployed on the servers. The access layer switches and their connected servers are collectively referred to as a rack (or block); a group of interconnected aggregation and access layer switches and their connected servers form a network module (or pod). For example, in Figure 6, access layer switch L0 and its connected servers H0 and H1 form a rack, while all devices within the first dashed box form a pod. As you can see, inter-Rack server communication needs to go through the aggregation layer switches. For example, communication between H0 and H2 needs to go through one or more switches S0 to S3. Inter-Pod server communication needs to go through the core layer switches. For example, communication between H0 and H8 needs to go through one or more switches C0 to C7.

[0130] Today's mainstream large-scale artificial intelligence (AI) models are trained using high-performance computing clusters (HPC) composed of distributed servers in data center networks. In this computing cluster, multiple servers equipped with high-performance GPUs (Graphics Processing Units) collaborate for training via high-bandwidth networks such as RoCEv2 (RDMA over Converged Ethernet) or InfiniBand (InfiniBand) networks. RDMA stands for Remote Direct Memory Access, or remote direct memory access. In a typical training task, the master server coordinates a group of servers for training based on a training framework, such as TensorFlow or PyTorch. The cluster training workload includes both computational and communication tasks. Computational tasks refer to the mathematical calculations performed on the AI ​​model by the server's GPU; communication tasks primarily involve synchronizing and coordinating the results of the calculations performed by each server, manifesting as data transmission between servers. Typically, communication tasks are implemented using collective communication operations in collective communication libraries (such as MPI, NCCL, and libfabric), such as Allreduce (global reduction), Alltoall, and Allgather. NCCL (Nvidia Collective Communication Library) is the most widely used collective communication library in the industry for large-scale model training, and Allreduce is the most basic and frequently used collective communication operation. In Allreduce, each node sends its own data, receives data from other nodes in the group, and generates a summary result. This embodiment primarily targets Allreduce operations, improving their throughput through data flow orchestration and scheduling.

[0131] In traditional training, Allreduce is generally implemented using a parameter server (PS) architecture. However, the completion time of the Allreduce operation in the PS architecture is proportional to the number of nodes, which means it is difficult to scale to large-scale networks. Therefore, the peer-to-peer architecture has emerged, which can effectively solve the scalability problem. In the peer-to-peer architecture, training mainly relies on distributed Allreduce collective operations to complete data synchronization. Implementations of distributed Allreduce operations include Ring Allreduce (ring reduction operation), Tree Allreduce (tree reduction operation), Halving-Doubling Allreduce (halving-doubling reduction operation), and their variants. Among them, Ring Allreduce is the most widely used Allreduce implementation. NCCL is based on the peer-to-peer architecture and uses distributed Allreduce operations for parameter synchronization.

[0132] In the logical implementation of Ring Allreduce, the master server strings all servers (a server typically has multiple GPUs) into a ring. Data is transmitted sequentially along the ring from one server to the next until all servers have received data from all other servers. The amount of data transmitted by each communication pair (for example, H1->H2) is the same. In this ring, the completion time of the Allreduce communication operation is equal to the transmission time of the slowest communication pair. In other words, a drop in the rate of any communication pair will become a bottleneck for the collective communication, slowing down the entire collective communication operation. In a real network, as shown in Figure 7, eight servers, H0 to H7, participate in Ring Allreduce. Servers H0, H1, H2, and H3 are directly connected to access layer switch L0, while servers H4, H5, H6, and H7 are directly connected to access layer switch L1. When processing based on Ring Allreduce, the optimal logical traffic ring formed by these servers is shown in Figure 8. In the Ring Allreduce optimal logical traffic ring, solid arrows represent intra-Rack traffic, and dashed arrows represent inter-Rack traffic. Data is transmitted from the direction indicated by the arrow to the direction pointed to by the arrow. The corresponding actual network traffic is shown in Figure 9. As can be seen, in this ring, six communications are intra-Rack, while the other two are inter-Rack. Intra-Rack communications, because they only traverse a single switch hop, are not subject to congestion (rate reduction). However, inter-Rack communications may experience rate reduction due to sharing the physical link with other data flows, which may also be part of this Allreduce operation. Inter-Rack traffic carries two risks: first, it can easily become congested with other traffic, resulting in rate reduction; second, latency can be significantly higher than intra-Rack communication. Therefore, during model training, it is important to minimize inter-Rack traffic within an Allreduce operation. Figure 8 and its corresponding network traffic diagram, Figure 9, illustrate the optimal traffic orchestration scenario. Inter-Rack traffic (dashed lines) has been minimized, with only one forward flow (H3->H4) and one reverse flow (H7->H0).

[0133] In actual large-scale production environments, especially in multi-tenant HPC training clusters provided by cloud vendors, cloud environment business deployment characteristics exist. That is, the server nodes provided to tenants may not be arranged continuously in physical locations, and the server list provided to tenants is often out of order. The main reasons are as follows:

[0134] Shielding the physical location from tenants; To prevent tenants from making difficult-to-meet server physical location requirements (such as requiring it to be within a rack), and to maximize server resource utilization, vendors need to present to tenants the impression that "all server resources have equal capabilities."

[0135] Server rentals are highly dynamic; because tenants can flexibly increase or decrease the number of servers they rent at any time, the locations of idle / available servers can be highly fragmented. Consequently, if existing tenants subsequently purchase additional servers or new tenants purchase servers, the physical locations of the servers they receive may be discontinuous. Furthermore, due to the chronological order of purchases, even if the servers are located consecutively, their order in the list may be out of order.

[0136] Server failure replacement: Due to the high performance and power requirements of HPC training scenarios, the failure rate of these servers (including GPU failure, network card failure, and whole machine failure, etc.) is relatively high, and scenarios where failed servers need to be replaced often occur. Due to time constraints, server replacements are all done using redundant hot standby replacement. That is, a few servers are deliberately reserved in the cluster and not sold. When a server fails, the redundant servers are given to tenants for use, while the failed server remains in place, waiting for physical repair or replacement. After the physical repair or replacement is completed, it will be sold again. In this way, for tenants, the newly added server will be directly added to the existing server list (added at the end). Therefore, the server list obtained by the tenant is often very messy in terms of physical distribution.

[0137] Existing collective communication libraries (such as NCCL) can perceive and optimize the GPU interconnect topology within a node through operating system queries, but they lack the ability to perceive the topological order between multiple server nodes. Currently, tenants typically receive a list of server nodes from a cloud operator, pass this list to the collective communication library, and then the library arranges the nodes into a traffic topology ring according to the order of the nodes in the list, performing a Ring Allreduce operation. However, as shown in Figures 8 and 9, this corresponds to the ideal scenario. In actual business scenarios, due to the limitations of the aforementioned cloud environment deployment capabilities, the node lists received by tenants are often out of order. In this case, the actual collective operation efficiency may be severely reduced (typically by 50%). It should be noted that although Ring Allreduce is the most commonly used Allreduce implementation, other Allreduce implementations are also used, such as Tree Allreduce and Halving-Doubling Allreduce. Similarly, these Allreduce implementations are also affected by the topological order of the servers, and this embodiment is also applicable to these implementations.

[0138] As can be seen, the performance of distributed Ring Allreduce relies heavily on excellent topology affinity and network capabilities. Under the optimal topology, as shown in Figure 8, machines in a rack only generate one set of upstream traffic and one set of downstream traffic. As shown in Figure 9, the four servers (H0-H3) on switch L0 in rack 1 generate only one set of upstream traffic (dashed line) passing through S1 to rack 2, while the four servers (H4-H7) on switch L1 in rack 2 also generate only one set of upstream traffic (dashed line) passing through S0 to rack 1. Note that the upstream traffic in rack 1 is also the downstream traffic in rack 2, and vice versa. Since each rack's single set of traffic originates from only one server, it does not cause congestion on any uplink of the access layer switch, naturally maintaining a very smooth network.

[0139] However, in actual training, if the topology received by the communication library is suboptimal, the resulting traffic loop is often not a minimum loop. Multiple flows may cross the aggregation layer, or even core layer switches. As shown in Figure 10, in the traffic logical loop for the eight servers H0-H7, as a single-tenant worst-case scenario, all inter-server communication is cross-layer traffic. As shown in Figure 11, in the single-tenant worst-case traffic topology ring, all inter-server communication crosses the access layer switches and is implemented through the aggregation layer switches. Due to the limitations of switch load balancing (the random nature of route hashing), these multiple upstream flows often cause a certain degree of congestion, resulting in reduced throughput between some node pairs. The characteristics of ring-shaped traffic mean that the overall system performance is limited by the connection with the minimum throughput (bottleneck). Therefore, congestion caused by a suboptimal traffic topology can severely impact the overall performance of the HPC system. For example, if the logical topology ring is arranged as shown in Figure 10, that is, the loop is H0-H4-H1-H5-H2-H6-H3-H7-H0, four groups of cross-rack upstream traffic (passing through the aggregation layer switch) and four groups of cross-rack downstream traffic will be generated in the network. The following two cases are discussed.

[0140] If the network bandwidth is convergent, meaning that the total downstream bandwidth of L0 is greater than the total upstream bandwidth, these multiple traffic flows will inevitably cause severe congestion, creating bottlenecks with poor throughput and significantly impairing system performance. As shown in Figure 11, four groups of upstream traffic from L0 enter the link to S0. Assuming the link bandwidth is 100G, since the maximum upstream traffic flow in each group is 100G, the total bandwidth of the four groups is 400G, and the bandwidth of the L0-S0 link is also only 100G, this results in an actual bandwidth of only 25G for each upstream traffic group (if congestion control is not fair, some traffic groups may have less than 25G), ultimately resulting in a total system throughput of only 25G (because the traffic topology is a ring, the throughput of the entire ring is limited by the throughput of the bottleneck point).

[0141] Even if network bandwidth is non-convergent, meaning that the total downstream bandwidth of L0 equals the total upstream bandwidth, imperfect load balancing can still lead to congestion on individual links, ultimately impacting system performance. As shown in Figure 11, assuming both the L0-S0 and L0-S1 connections are two 100G links, due to the imperfect load balancing hashing of the switch, there is a high probability (90%) that two or more of the four upstream traffic flows will enter the same 100G link. In this case, the throughput of the entire system will drop to 50G or even lower.

[0142] It's worth noting that the above discussion only applies to the small network shown in Figure 6. In actual large-scale networks, performance is more random, and unoptimized system performance may deviate significantly from optimal performance. For example, in a real-world production network, a rack / block typically has 32 servers. In the worst-case scenario, a rack will send 32 sets of cross-rack traffic. With this amount of traffic, there's a high probability that at least two sets of traffic will be congested on a single port in the upstream routing hash. In this case, the actual throughput of these congested flows will drop to 50% or less of the ideal throughput. In a Ring Allreduce operation, any bottleneck in the ring will drag down the throughput of the entire operation, causing the completion time to increase by at least twice the ideal time. For these reasons, the server node lists presented to tenants by vendors don't guarantee topological affinity. In practice, node names are often composed of random letters and numbers, and no topological affinity can be discerned from the node names in the list. Furthermore, the node IP (Internet Protocol) addresses do not contain information that can identify their physical locations. For tenants, if they directly use the node list provided by the cloud provider as the input list for the collective communication library, they may experience performance lower than expected. Furthermore, due to the randomness of load balancing within the network, even if the network topology is designed to be non-convergent in bandwidth (i.e., the downlink and uplink bandwidth of the LA are the same), the HPC performance achieved by tenants will be random (inconsistent) and difficult to predict.

[0143] Moreover, the above scenarios are all for single-tenant scenarios. In cloud deployment environments, there are multi-tenants, that is, servers of multiple tenants are deployed in the same network. In a multi-tenant scenario, even in the most ideal case, there will be multiple sets of cross-Rack traffic in the network. If the traffic topology of multiple tenants is not optimized, there will be more cross-Rack traffic, which will bring more performance uncertainty to model training, and in turn cause a significant impact on the SLA (Service-Level Agreement) commitments made by the manufacturer to the tenants. For example, as shown in Figure 12, in a multi-tenant server mixed deployment scenario, if there are two tenants' servers, the servers rented by tenant 1 include the servers H0, H2, H4, and H6 with filled slashes, and the servers rented by tenant 2 include the unfilled servers H1, H3, H5, and H7. Ideally, if each tenant's traffic topology is optimized, their logical traffic topology and network traffic are shown in Figures 13 and 14, respectively. The dashed lines represent the traffic topology between Tenant 1's servers H0, H2, H4, and H6, while the solid lines represent the traffic topology between Tenant 2's servers H1, H3, H5, and H7. Each access layer switch has two groups of upstream traffic, and the probability of congestion for these two groups of traffic is relatively low. However, if each tenant's traffic topology is not optimized, as shown in Figures 15 and 16, each access layer switch has four groups of upstream traffic. The probability of congestion for these four groups of traffic is much higher. Furthermore, the latency of Ring Allreduce operations is significantly higher with an unoptimized topology.

[0144] In a real-world network, a rack contains eight leaf switches and 32 servers, each with eight network adapters. A leaf switch connects to 64 ports downstream and 64 ports upstream. In the extreme case, a leaf switch can have 64 cross-rack traffic flows upstream. By analyzing the relationship between the number of cross-rack flows and the probability of congestion, Figure 17 shows the probability of at least one LA uplink congestion on the access layer switch. The horizontal axis represents the number of uplink flows on the access layer switch, and the vertical axis represents the probability. In the relationship between the number of uplink flows on a leaf switch and the probability of congestion, the dashed line represents the unoptimized / random curve, while the solid line represents the probability curve after topology optimization orchestration when a single tenant uses Ring Allreduce. As can be seen, when the number of uplink flows on a leaf exceeds 20, even if the switch has 64 uplinks, the probability of at least one link being congested exceeds 90%. When the number of upstream flows exceeds 30, congestion (at least one link) is almost certain to occur (with a probability close to 100%).

[0145] In view of the above practical reasons, this embodiment optimizes and arranges the tenant's server node list (as shown in Figure 10 to Figure 8). The optimized traffic topology can minimize the upstream traffic of each rack (as shown in Figure 8), thereby minimizing the congestion of the upstream traffic of the access layer switch, improving network connectivity, and maximizing the tenant's network throughput. In particular, when there is only one tenant in the rack, this embodiment can ensure that the entire rack has only one set of upstream traffic, and the upstream direction of the access layer switch will not cause any congestion, and the tenant throughput can reach an ideal level.

[0146] This embodiment can assist the use of collective communication libraries in large model training in the form of components. A large model training often involves multiple collective operations at the same time, and the servers for multiple collective operations generally do not overlap. For example, in 3-D (Three-Dimensional) parallel training (model parallelism, pipeline parallelism, and data parallelism), all GPUs will participate in the Allreduce operation of a data parallel plane to synchronize the training results of different data parallel planes, with a total of dozens of Allreduce operations. Each collective communication operation has a master server, which initiates the entire operation. In actual training, if this solution requires the master server of each collective communication operation to initiate a topology optimization orchestration request (the request is completed by the topology optimization agent component on the server), after receiving the request, the controller will optimize the orchestration of the server list in the request to increase the affinity of the collective communication, reduce the latency of the communication operation and network congestion, and improve the throughput of the collective communication, thereby reducing the training time of large model training.

[0147] The collective communication processing method provided in this embodiment is implemented in the form of an NCCL plugin on the server side when a large-scale AI training cluster, such as an HCC (High-Performance Computing Cluster) product, is implemented. The controller is implemented by a dedicated server and can be connected to a corresponding network topology database. As shown in Figure 18, for different collective operations, each collective operation is implemented by multiple servers, including a master server and several slave servers. The master server can be provided with a collective communication library, which can be specifically NCCL. The master server can send the original node list to the controller, and the controller returns the optimal node list that has been updated after the original node list is arranged, so that the master server can initiate the corresponding collective operation based on the received optimal node list.

[0148] Specifically, in actual business, tenants will receive a set of original server node lists after purchasing a server. During model training, tenants use training frameworks (such as TensorFlow or PyTorch, etc.) to allocate server resources. When performing a certain collective operation, the training framework will pass part of the node list to the collective communication library (such as NCCL or MPI, etc.). Generally, in model training, not all servers are used for the same collective operation. Each collective operation is only a part of the training. There are many kinds of collective operations in the entire training, and each collective operation will only use some servers. When the training framework calls the collective communication library to complete the collective operation, it will pass the server list (operation node list) used for the operation to the collective communication library. As shown in Figure 19, in step 0, the training framework of the main server obtains the entire original node list and passes the operation node list as the original node list to the collective communication library.

[0149] Furthermore, the composition of this embodiment mainly includes: a topology optimization agent on the server side and a centralized controller that provides topological sorting capabilities. The controller can be a physically independent server or a logical module on an existing server. The agent and the controller communicate through the network in a "request-response" manner. In this embodiment, the collective communication library can be slightly modified to add a request topology optimization interface. In some communication libraries, it is possible to modify its plug-in components instead of the communication library itself, such as the NCCL built-in plugin function interface, which only needs to be added to the NCCL plugin.

[0150] As shown in step ① in Figure 19, when the collective communication library of the main server receives the collective communication operation command, it will first request topology orchestration optimization from the topology optimization agent of this solution; step ②, after receiving the request from the collective communication library, the topology optimization agent will request a topology sorting request from the controller through the network; step ③, after receiving the request, the topology sorting engine on the controller will first query the topology database; step ④, the controller sorts according to the topology information of each server node, and then sends the sorted server list back to the topology optimization agent; step ⑤, after receiving the topology orchestration optimization response, the topology optimization agent will return the result to the collective communication library of the main server, and the collective communication library will use the orchestrated server list to initiate collective communication.

[0151] Furthermore, for topological clustering, or affinity-aware processing, the topological sorting engine first queries the topology database for the topology information of each server in the operating server list. The format of the topology information is similar to {HostId:ClusterId,PodId,RackId}. The topology information can be queried in the topology database using the server's IP address or fixed asset number. As shown in Figure 20, each server node can be divided into Cluster, Pod, and Rack. The topology information of server node H16 can be expressed as {H16:1,2,4}.

[0152] After querying the topology information of all servers, the topological sorting engine will hierarchically cluster the servers. The specific steps are as follows:

[0153] a) Cluster all servers based on their ClusterId, i.e., cluster servers with the same ClusterId together (arrange them adjacently). This involves traversing all servers and checking their ClusterIds. If a cluster with the same ClusterId exists, the server is added to that cluster; otherwise, a new cluster with the same ClusterId is created and the server is added to that cluster.

[0154] b) Within each cluster with the same ClusterId, cluster servers based on their PodId, grouping them together (arranging them adjacently). This involves iterating through all servers in each Cluster class, checking their PodIds, and adding the server to a cluster if the class with the PodId already exists. Otherwise, a new class with the PodId is created and added to the cluster.

[0155] c) Within each cluster with the same ClusterId and PodId, cluster servers based on their RackId, grouping them together (arranging them adjacently). This involves iterating through all servers in each Pod class, checking their RackIds, and adding the server to a class if the class with the RackId already exists in the cluster. Otherwise, a new class with the RackId is created and added to the class. At this point, all servers participating in the collective communication operation have been clustered, effectively achieving affinity awareness.

[0156] As shown in Figure 20, the list of servers participating in a collective communication operation may be: [H11, H25, H14, H21, H2, H6, H13, H0, H22, H30, H16, H5, H29, H23, H7, H18]. Then, as shown in Figure 21, this is the clustering result for the list of servers participating in a collective communication operation in Figure 20. Specifically, after clustering by ClusterId, two Cluster classes are obtained, namely {0: [H11, H14, H2, H6, H13, H0, H5, H7]} and {1: [H25, H21, H22, H30, H16, H29, H23, H18]}. After clustering by PodId, we get four Pod classes: {0:[H2,H6,H0,H5,H7]}, {1,[H11,H14,H13]}, {2:[H21,H22,H16,H23,H18]}, and {3:[H25,H30,H29]}. After clustering by RackId, we get eight Rack classes: {0:[H2,H0]}, {1:[H6,H5,H7]}, {2:[H11]}, {3:[H14,H13]}, {4:[H16,H18]}, {5:[H21,H22,H23]}, {6:[H25]}, and {7:[H30,H29]}.

[0157] Furthermore, for node orchestration optimization, after obtaining the clustering results, we need to optimize the topology orchestration based on the results. The specific method is to perform a depth-first search / traversal on the clustering results. That is, in the cluster tree, we search for the Pods under each Cluster, and then search for the Racks under each Pod. The searched nodes are sequentially placed in the optimization list L.

[0158] For example, for the clustering tree shown in Figure 21:

[0159] a) First, Cluster 0 is searched. Since it has not yet reached a leaf node, Pod 0 is searched. Since it has not yet reached a leaf node, Rack 0 is searched. Rack 0 is already a leaf node, and its server (H2, H0) is added to the optimization list L. At this point, L = [H2, H0].

[0160] b) Then search for other racks under Pod 0, find Rack 1, and add its servers to the optimization list L. At this time, L = [H2, H0, H6, H5, H7].

[0161] c) At this point, the leaves of Pod 0 have been searched, and the search continues for the other Pods in Cluster 0, namely Pod 1. Rack 2 below it is searched, and its servers are placed in L, where L = [H2, H0, H6, H5, H7, H11].

[0162] d) Then search for Rack 3 under Pod 1 and put its servers into L, where L = [H2, H0, H6, H5, H7, H11, H14, H13].

[0163] e) At this point, Cluster 0 has been searched, and Cluster 1 is searched next. This process continues until all racks have been searched. The final optimized list is L = [H2, H0, H6, H5, H7, H11, H14, H13, H16, H18, H21, H22, H23, H25, H30, H29].

[0164] Note that the search order in the above process is not unique. For example, you can search Cluster 1 first and then Cluster 0. For another example, in Pod 0, you can search Rack 1 first and then Rack 0. Furthermore, the order of servers within a Rack is not restricted. For example, in Rack 0, scheduling H0 or H2 first is the same; the final logical effect is the same.

[0165] When the orchestrated list is used for the Ring Allreduce operation, the traffic topology formed is shown in Figure 22, where the different heights of the lines represent the switch layers that the group of traffic crosses. For example, H13->H16, H29->H2 all cross the backbone layer switches, and H7->H11, H23->H25 all cross the core layer switches. It can be seen that the optimized orchestrated traffic has minimized the number of flows crossing the aggregation layer, core layer, and backbone layer switches. This can minimize the probability of these flows being congested with themselves and other flows, thereby improving training throughput. It should be noted that the optimized orchestrated topology of this embodiment can also be used for other implementations of Allreduce, including Tree-Allreduce, Halving-Doubling, and other implementations.

[0166] In addition, the collective communication processing method provided in this embodiment can also be changed with respect to the search starting point. In this embodiment, the search starts from the 0th Rack of the 0th Pod of the 0th Cluster. As an example of a change, the search can also start from the 4th Rack of the 2nd Pod of the 1st Cluster. The final orchestration optimized topology rings are actually the same, just rotated, that is, the starting points of the server nodes are different. The order of nodes in the same Rack can also be changed. In this embodiment, multiple nodes in the same Rack are arranged according to the order in which they are in the original operation node list, and can actually be rearranged arbitrarily.

[0167] The collective communication processing method provided in this embodiment optimizes the traffic scheduling of collective operations in language large model training (AI HPC), including the traffic scheduling of the most widely used Allreduce operation. This embodiment uses centralized topological sorting technology and a request-response-based architecture to optimize the server list provided to tenants by cloud HPC service providers, improve the affinity of collective operations, and thus optimize the traffic scheduling and throughput performance of collective communications. This embodiment can eliminate HPC traffic congestion in the network in a single-tenant scenario; it can also minimize network congestion in a multi-tenant scenario, thereby maximizing the throughput and efficiency of the HPC cluster, while significantly improving the smoothness and stability of the network. Compared to unoptimized scenarios, this embodiment can provide up to 70% throughput improvement.

[0168] The collective communication processing method provided in this embodiment can minimize the latency of collective communication operations. Specifically, in Ring Allreduce operations, optimized topology orchestration can reduce communication latency by over 50% compared to unoptimized orchestration. The collective communication processing method provided in this embodiment can also minimize network bandwidth contention (network congestion), improve network connectivity, eliminate bottlenecks in the collective communication traffic loop, and maximize the throughput of collective operations. Specifically, if there is only one tenant in the network, this embodiment can completely eliminate congestion and achieve ideal network throughput. As shown in Figure 23, in a single-tenant scenario on a 100Gb network, 10 servers distributed across two racks are used to perform Ring Allreduce operations. After running both solutions 200 times on a 100Gb network, the optimized topology can achieve full system throughput, with a net throughput (Busbw) of approximately 12GB. In contrast, the unoptimized traffic topology often only achieves a throughput of around 7-8GB. If there are multiple tenants, the unoptimized traffic topology may suffer even greater throughput loss due to network congestion.

[0169] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0170] Based on the same inventive concept, embodiments of the present application also provide a collective communication processing device for implementing the collective communication processing method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more collective communication processing device embodiments provided below can be found in the above-mentioned limitations on the collective communication processing method and will not be repeated here.

[0171] In an exemplary embodiment, as shown in FIG24 , a collective communication processing apparatus 2400 is provided, comprising: a node orchestration request acquisition module 2402, a topology information acquisition module 2404, a node orchestration update module 2406, and a collective operation processing module 2408, wherein:

[0172] A node orchestration request acquisition module 2402 is configured to acquire a node orchestration request for a target collective operation in collective communication. The node orchestration request carries an original node list, which includes server nodes selected to execute the target collective operation.

[0173] A topology information acquisition module 2404 is configured to acquire topology information of each server node. The topology information is used to describe the switch nodes connected to the server nodes at each level of the node communication network. The server nodes communicate with each other based on the switch nodes connected to each level of the node communication network.

[0174] The node arrangement updating module 2406 is used to arrange and update the arrangement order of each server node step by step according to the hierarchy in the node communication network based on each topology information to obtain an arrangement node list; and

[0175] The collective operation processing module 2408 is configured to perform collective communication for a target collective operation through each server node according to the arrangement order of each server node in the orchestration node list.

[0176] In one embodiment, the node orchestration update module 2406 is further configured to hierarchically cluster each server node according to the hierarchy in the node communication network based on each topology information to obtain a topological clustering result for each server node; and to search for nodes according to the hierarchy in the node communication network based on each topological clustering result to obtain an orchestration node list, wherein the arrangement order of each server node in the orchestration node list is the same as the hierarchical order of the node search.

[0177] In one embodiment, the node orchestration update module 2406 is further used to hierarchically cluster each server node step by step based on each topology information according to the hierarchical dimension from high to low in the node communication network to obtain a topological clustering result of each server node; in the node communication network, high-level switching nodes are used to support cross-node communication between server nodes connected to low-level switching nodes.

[0178] In one embodiment, the layers in the node communication network include a core layer, a convergence layer, and an access layer from high to low; the node orchestration update module 2406 is further used to cluster the server nodes according to the layer dimension of the core layer based on the topology information to obtain the core layer clustering results of the server nodes; cluster the server nodes according to the layer dimension of the convergence layer based on the core layer clustering results to obtain the convergence layer clustering results of the server nodes; and cluster the server nodes according to the layer dimension of the access layer based on the convergence layer clustering results to obtain the topology clustering results of the server nodes.

[0179] In one embodiment, the node orchestration update module 2406 is further used to perform node searches for each server node step by step based on the topological clustering results and in accordance with the hierarchical dimensions from high to low in the node communication network to obtain an orchestration node list; in the node communication network, high-level switching nodes are used to support cross-node communication between server nodes connected to low-level switching nodes.

[0180] In one embodiment, the levels in the node communication network include, from high to low, a core layer, a convergence layer, and an access layer; the node orchestration update module 2406 is further used to perform node searches for each server node step by step based on the hierarchical dimensions from high to low in the node communication network based on each topological clustering result, to obtain an orchestration node list, including: searching according to the hierarchical dimension of the core layer based on each topological clustering result to obtain a core layer search result; searching according to the hierarchical dimension of the convergence layer based on the core layer search result to obtain a convergence layer search result; searching according to the hierarchical dimension of the access layer based on the convergence layer search result to obtain an access layer search result; and obtaining an orchestration node list based on the core layer search result, the convergence layer search result, and the access layer search result.

[0181] In one embodiment, the node orchestration request acquisition module 2402 is further configured to determine a collective communication library in the collective communication; and obtain a node orchestration request for a target collective operation in the collective communication based on a communication interface with the collective communication library.

[0182] In one embodiment, the topology information acquisition module 2404 is further configured to determine a topology database associated with the node communication network; and query the topology information of each server node from the topology database according to the original node list.

[0183] In one embodiment, the node orchestration request acquisition module 2402 is further configured to acquire a node orchestration request from a target server node, where the target server node is determined from among the server nodes selected for performing the target collective operation in collective communication. The collective operation processing module 2408 is further configured to send an orchestration node list to the target server node, where the orchestration node list is configured to instruct the target server node to perform collective communication according to the target collective operation through each server node in the order in which the server nodes are arranged in the orchestration node list.

[0184] In one embodiment, the system further includes a trigger determination module configured to execute the step of obtaining a node orchestration request for the target set operation in the collective communication when each server node for the target set operation meets an orchestration update trigger condition.

[0185] In an exemplary embodiment, as shown in FIG25 , a collective communication processing apparatus 2500 is provided, comprising: an original node list acquisition module 2502, a node orchestration request module 2504, and a collective operation processing module 2506, wherein:

[0186] The original node list acquisition module 2502 is used to determine the original node list when triggering the target collective operation in the collective communication, where the original node list includes each server node selected to perform the target collective operation;

[0187] A node orchestration request module 2504 is configured to send a node orchestration request generated based on the original node list to the server, wherein the node orchestration request is used to instruct the server to return an orchestration node list. The orchestration node list is based on the topology information of each server node and is updated by orchestrating the arrangement order of each server node in the original node list step by step according to the hierarchy in the node communication network; and

[0188] The collective operation processing module 2506 is configured to receive the orchestration node list and perform collective communication according to the target collective operation through each server node in the arrangement order of each server node in the orchestration node list.

[0189] Each module in the collective communication processing device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0190] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal, and its internal structure diagram may be shown in FIG26 . The computer device includes a processor, a memory, an input / output (I / O) interface, and a communication interface. The processor, the memory, and the I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer device is used to store collective communication processing data. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a collective communication processing method. Those skilled in the art will understand that the structure shown in Figure 26 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0191] In one embodiment, a computer device is further provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps in the above-mentioned method embodiments when executing the computer-readable instructions.

[0192] In one embodiment, a computer-readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0193] In one embodiment, a computer program product is provided, comprising computer-readable instructions, which implement the steps in the above-mentioned method embodiments when executed by a processor.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0195] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0196] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent of this application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of this application, several variations and improvements can be made, which all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be based on the attached claims.

Claims

1. A collective communication processing method, executed by a second server, the method comprising: Obtaining a node orchestration request for a target collective operation in collective communication, the node orchestration request carrying an original node list, the original node list including each server node selected for executing the target collective operation; Acquiring topology information of each of the server nodes, the topology information being used to describe the switching nodes connected to the server nodes at each level of the node communication network, and the server nodes communicating with each other based on the switching nodes connected at each level of the node communication network; Based on the topology information, the arrangement order of each server node is updated step by step according to the hierarchy in the node communication network to obtain an arrangement node list; and According to the arrangement order of the server nodes in the orchestration node list, collective communication is performed for the target collective operation through the server nodes.

2. The method according to claim 1, wherein, based on the topology information, the arrangement order of each server node is updated step by step according to the hierarchy in the node communication network to obtain an arrangement node list, comprising: Based on each of the topology information, hierarchically clustering each of the server nodes according to the hierarchy in the node communication network, to obtain a topological clustering result for each of the server nodes; and Based on the topological clustering results, node searches are performed level by level according to the hierarchy in the node communication network to obtain an orchestration node list, wherein the arrangement order of the server nodes in the orchestration node list is the same as the hierarchical order of the node search.

3. The method according to claim 2, wherein the server nodes are hierarchically clustered based on the topology information and according to the hierarchy in the node communication network to obtain a topological clustering result for each server node, comprising: Based on each of the topology information, hierarchically clustering each of the server nodes step by step according to the hierarchical dimensions from high to low in the node communication network to obtain a topological clustering result of each of the server nodes; In the node communication network, high-level switching nodes are used to support cross-node communication between server nodes connected to low-level switching nodes.

4. The method according to claim 3, wherein the layers in the node communication network include, from high to low, a core layer, a convergence layer, and an access layer; The topology information is based on each server node, and the server nodes are hierarchically clustered step by step according to the hierarchical dimensions from high to low in the node communication network to obtain a topological clustering result of each server node, including: Clustering the server nodes according to the hierarchical dimension of the core layer based on the topology information to obtain a core layer clustering result of the server nodes; Based on the core layer clustering results, clustering the server nodes according to the hierarchical dimension of the convergence layer to obtain the convergence layer clustering results of the server nodes; and Based on the clustering results of each convergence layer and according to the hierarchical dimension of the access layer, the server nodes are clustered to obtain a topological clustering result of each server node.

5. The method according to claim 2, wherein the step of searching for nodes level by level in the node communication network based on the topological clustering results to obtain an orchestration node list comprises: Based on the topological clustering results, performing node searches for each server node step by step according to the hierarchical dimensions from high to low in the node communication network to obtain an orchestration node list; In the node communication network, high-level switching nodes are used to support cross-node communication between server nodes connected to low-level switching nodes.

6. The method according to claim 5, wherein the layers in the node communication network include, from high to low, a core layer, a convergence layer, and an access layer; Based on the topological clustering results, according to the hierarchical dimensions from high to low in the node communication network, node searches are performed for each server node step by step to obtain an orchestration node list, including: Based on each of the topological clustering results, a search is performed according to the hierarchical dimension of the core layer to obtain a core layer search result; Searching the core layer search results according to the hierarchical dimension of the convergence layer to obtain the convergence layer search results; Searching according to the hierarchical dimension of the access layer based on the convergence layer search results to obtain access layer search results; and An orchestration node list is obtained according to the core layer search results, the convergence layer search results, and the access layer search results.

7. The method according to claim 1, wherein obtaining a node orchestration request for a target collective operation in collective communication comprises: determining a collective communication library in the collective communication; and Based on the communication interface with the collective communication library, a node orchestration request for a target collective operation in the collective communication is obtained.

8. The method according to claim 1, wherein obtaining topology information of each of the server nodes comprises: Determine the topological database of the node communication network association; and According to the original node list, the topology information of each server node is obtained by querying from the topology database.

9. The method according to claim 1, wherein obtaining a node orchestration request for a target collective operation in collective communication comprises: Obtaining a node orchestration request from a target server node, wherein the target server node is determined from among the server nodes selected for performing a target collective operation in the collective communication; The performing collective communication for the target collective operation through each server node according to the arrangement order of each server node in the orchestration node list includes: The orchestration node list is sent to the target server node, where the orchestration node list is used to instruct the target server node to perform collective communication according to the target collective operation through each server node in the arrangement order of the server nodes in the orchestration node list.

10. The method according to any one of claims 1 to 9, further comprising: When each server node operating on the target set meets an orchestration update triggering condition, the step of obtaining a node orchestration request for the target set operating in the collective communication is performed.

11. A collective communication processing method, executed by a first server, the method comprising: When a target collective operation in collective communication is triggered, an original node list is determined, wherein the original node list includes each server node selected for performing the target collective operation; Sending a node orchestration request generated according to the original node list to the server, wherein the node orchestration request is used to instruct the server to return an orchestration node list, wherein the orchestration node list is obtained by orchestrating and updating the arrangement order of each server node in the original node list based on topology information of each server node and according to the hierarchy in the node communication network; and An orchestration node list is received, and collective communication is performed according to the target collective operation through each server node in the arrangement order of each server node in the orchestration node list.

12. A collective communication processing device, comprising: a node orchestration request acquisition module, configured to acquire a node orchestration request for a target collective operation in collective communication, wherein the node orchestration request carries an original node list, and the original node list includes each server node selected to execute the target collective operation; a topology information acquisition module, configured to acquire topology information of each of the server nodes, wherein the topology information is used to describe the switching nodes connected to the server nodes at each level of the node communication network, and wherein the server nodes communicate with each other based on the switching nodes connected at each level of the node communication network; A node arrangement updating module is configured to arrange and update the arrangement order of each server node step by step according to the hierarchy in the node communication network based on each of the topology information to obtain an arrangement node list; and The collective operation processing module is configured to perform collective communication for the target collective operation through each of the server nodes according to the arrangement order of the server nodes in the orchestration node list.

13. A collective communication processing device, comprising: An original node list acquisition module is used to determine an original node list when a target collective operation in collective communication is triggered, wherein the original node list includes each server node selected for executing the target collective operation; a node orchestration request module, configured to send a node orchestration request generated based on the original node list to a server, wherein the node orchestration request is used to instruct the server to return an orchestration node list, wherein the orchestration node list is obtained by orchestrating and updating the arrangement order of each server node in the original node list based on topology information of each server node and according to the hierarchy in the node communication network; and The collective operation processing module is configured to receive an orchestration node list and perform collective communication according to the target collective operation through each server node in the arrangement order of each server node in the orchestration node list.

14. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method according to any one of claims 1 to 11 when executing the computer-readable instructions.

15. A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

16. A computer program product comprising computer-readable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images