Communication Scheduling Method, Electronic Device, and Medium for Distributed Large Model Training

Abstract

The present invention discloses a communication scheduling method, an electronic device, and a medium for distributed large model training, including: obtaining the scale, degree of the server cluster, and the transmission demand ratio of AllReduce and MP traffic, splitting them into the degrees of the AllReduce traffic and MP traffic sub-topologies according to the two types of traffic demand ratios, constructing the AllReduce sub-topology and MP sub-topology based on this, combining them to obtain a topology graph, and implementing the physical topology using an optical switch; when the traffic changes, obtaining all distributed large model training task arrays, link arrays, and candidate placement location arrays; constructing an affinity graph corresponding to each candidate placement location according to the topology graph, calculating the compatibility scores of all links in the affinity graph to obtain the optimal placement location; calculating the delays of all connection sub-graphs in the affinity graph corresponding to the optimal placement location, and taking the sum thereof as the total delay, with the minimum total delay as the optimization goal; performing distributed large model training on the physical topology according to the optimal placement location and the total delay.

Description

Technical Field

[0001] The present invention belongs to the field of network traffic scheduling in computer distributed large model training, and particularly relates to a communication scheduling method, an electronic device, and a medium for distributed large model training. Background Art

[0002] In the training environment of distributed deep learning, as the scale and complexity of the model continue to increase, the communication requirements and traffic patterns during the training process become more dynamically complex. Especially when training complex models such as graph neural networks (GNNs) and mixture of experts models (MoEs), different computing tasks and communication tasks will exhibit significant traffic change characteristics in different iterations of the training. Traditional static network topologies often lack flexibility in coping with these dynamic changes and cannot adapt to frequent communication requirement adjustments, resulting in low network resource utilization, high communication overhead, and limited model training speed.

[0003] To solve this problem, existing research has introduced dynamic network topology optimization to improve communication efficiency by reconstructing the network at runtime. However, frequent topology reconstruction may introduce additional overhead, involving significant reconstruction costs for switches and links. During the topology change process, for model parallel and data parallel tasks such as AllReduce synchronization operations, this latency will cause performance losses. Summary of the Invention

[0004] The purpose of the present invention is to provide a communication scheduling method, an electronic device, and a medium for distributed large model training in view of the deficiencies of the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a communication scheduling method for distributed large model training, the method comprising:

[0006] Obtaining the number of servers for distributed large model training, the degree of each server, the transmission traffic requirements of AllReduce traffic, and the transmission traffic requirements of MP traffic;

[0007] Dividing the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission requirements of AllReduce traffic and MP traffic;

[0008] Constructing an AllReduce sub-topology and an MP sub-topology respectively according to the transmission requirements and degrees of AllReduce traffic and MP traffic; combining the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; configuring an optical switch according to the topology graph to implement a physical topology;

[0009] Monitor the traffic changes of all servers during the distributed large model training process. When the traffic changes, obtain all distributed large model training task arrays, link arrays, and candidate placement location arrays. Among them, the candidate placement location indicates placing a certain task on a certain server.

[0010] Traverse each candidate placement location. For each candidate placement location, construct a corresponding affinity graph according to the topology graph, calculate the compatibility scores of all links in the affinity graph under the condition that the bandwidth demand satisfies the link capacity constraint, and take the candidate placement location with the highest compatibility score as the best placement location. Calculate the latency of all connected subgraphs in the affinity graph corresponding to the best placement location, and take the sum of the latencies of all connected subgraphs as the total latency, with the goal of minimizing the total latency. Perform distributed large model training on the physical topology according to the best placement location and the total latency.

[0011] In a second aspect, an embodiment of the present invention provides a communication scheduling system for distributed large model training. The system includes:

[0012] A topology reconstruction component, which is used to obtain the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of AllReduce traffic, and the transmission traffic demand of MP traffic; divide the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission demands of AllReduce traffic and MP traffic; construct an AllReduce sub-topology and an MP sub-topology respectively according to the transmission demands of AllReduce traffic and MP traffic and the degrees; combine the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; configure an optical switch according to the topology graph to implement a physical topology.

[0013] A traffic awareness component, which is used to monitor the traffic changes of all servers during the distributed large model training process. When the traffic changes, obtain all distributed large model training task arrays, link arrays, and candidate placement location arrays. Among them, the candidate placement location indicates placing a certain task on a certain server. Traverse each candidate placement location. For each candidate placement location, construct a corresponding affinity graph according to the topology graph, calculate the compatibility scores of all links in the affinity graph under the condition that the bandwidth demand satisfies the link capacity constraint, and take the candidate placement location with the highest compatibility score as the best placement location. Calculate the latency of all connected subgraphs in the affinity graph corresponding to the best placement location, and take the sum of the latencies of all connected subgraphs as the total latency, with the goal of minimizing the total latency. Perform distributed large model training on the physical topology according to the best placement location and the total latency.

[0014] In a second aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned communication scheduling method for distributed large model training.

[0015] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned communication scheduling method for distributed large model training is implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the above-mentioned communication scheduling method for distributed large model training is implemented.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] The present invention provides a communication scheduling method for distributed large model training, which is used for distributed large model training tasks and constructs a topology graph based on transmission traffic requirements, and configures an optical switch according to the topology graph to implement a physical topology; at the same time, monitors the traffic changes of all servers during the distributed large model training process, adjusts the latency of jobs, reduces network congestion and link bottlenecks, thereby reducing the need for physical topology reconstruction. The present invention combines intelligent scheduling and dynamic topology to minimize both the communication overhead and the topology reconstruction cost in distributed large model training, and optimizes the communication efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0020] Figure 1 It is a schematic diagram of the communication scheduling method for distributed large model training provided by an embodiment of the present invention;

[0021] Figure 2 It is a flowchart of constructing a topology graph provided by an embodiment of the present invention;

[0022] Figure 3 It is a flowchart of traffic awareness and task scheduling provided by an embodiment of the present invention;

[0023] Figure 4 It is a schematic diagram of a sharded cluster system provided by an embodiment of the present invention;

[0024] Figure 5 It is a logical schematic diagram of a bipartite affinity diagram provided by an embodiment of the present invention;

[0025] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. Specific implementation manners

[0026] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that, without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0028] As Figure 1 shown, an embodiment of the present invention provides a communication scheduling method for distributed large model training, and the method includes the following steps:

[0029] Step S1, obtain the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of AllReduce traffic, and the transmission traffic demand of MP traffic.

[0030] Specifically, in this example, obtain the task information, load pattern, and parallelization strategy of the distributed large model training task; among them, the task information of the distributed large model training task includes the task type, computing requirements, and data scale; the load patterns of the distributed large model training task include: compute-intensive, data-intensive; the parallelization strategies of the distributed large model training task include: data parallelism, model parallelism;

[0031] Obtain the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of AllReduce traffic, and the transmission traffic demand of MP traffic according to the task information, load pattern, and parallelization strategy of the distributed large model training task.

[0032] It should be noted that the MP traffic is related to model parallelism and includes the activation values and gradient data calculated during forward calculation and backpropagation. The traffic characteristics are scattered and the traffic is small. The AllReduce traffic is generated during both data parallelism and traffic parallelism. It is mainly used to synchronize model parameters between different aggregators during the global reduction (AllReduce) operation after a training sample is trained.

[0033] Step S2: Divide the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission requirements of the AllReduce traffic and the MP traffic; construct the AllReduce sub-topology and the MP sub-topology according to the transmission requirements and degrees of the AllReduce traffic and the MP traffic respectively; combine the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; configure an optical switch according to the topology graph to implement a physical topology.

[0034] Further, as Figure 2 shown, the specific steps of step S2 are as follows:

[0035] Step S201: Divide the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission requirements of the AllReduce traffic and the MP traffic.

[0036] Specifically, the expression is as follows:

[0037] T = T MP + T AllReduce

[0038] In the formula, T represents the total traffic, T MP represents the transmission requirement of the MP traffic, and T AllReduce represents the transmission requirement of the AllReduce traffic.

[0039] d A = (T AllReduce / T) × d

[0040] d MP = (T MP / T) × d

[0041] In the formula, d represents the degree of the server, d A represents the degree assigned to the AllReduce traffic sub-topology, and d MP represents the degree assigned to the MP traffic sub-topology.

[0042] Step S202: Construct the AllReduce sub-topology according to the transmission requirement and degree of the AllReduce traffic.

[0043] For each AllReduce group, allocate the corresponding degree for each AllReduce group according to the traffic corresponding to each AllReduce group;

[0044] Take the server as a node and use an empty set as the initialized first candidate permutation set; calculate the Euler's totient function for the degree corresponding to each server, and the result output by the Euler's totient function represents different connection methods between servers; fill the result output by the Euler's totient function into the second candidate permutation set.

[0045] In response to the number of servers, the degree corresponding to each AllReduce group, and the second candidate permutation set corresponding to each AllReduce group, construct the first candidate permutation corresponding to each AllReduce group, and iteratively optimize the first candidate permutation through the selection permutation combination method to obtain the topology corresponding to each AllReduce group.

[0046] Merge the topologies corresponding to each AllReduce group to obtain the AllReduce sub-topology.

[0047] Exemplarily, in this example, take the k-th AllReduce group as an example to elaborate on the specific process of constructing the AllReduce sub-topology; including:

[0048] Obtain the server set, take the server as a node, mark each node, and use an empty set as the initialized first candidate permutation set G AllReduce ={}.

[0049] For the degree of each server node, calculate its Euler's totient function. The result of the Euler's totient function is the number of positive integers that are relatively prime to this degree, that is, the possible connection combinations between servers, that is, the different permutation methods between servers, denoted as the second candidate permutation set P k .

[0050] In response to the number of servers n, the degree d corresponding to each AllReduce group k , and the second candidate permutation set P corresponding to each AllReduce group k , initialize an empty set as the first candidate permutation G corresponding to the k-th AllReduce group k .

[0051] Take the first element P k in the second candidate permutation set P k [0] and assign it to q. q represents the smallest permutation in the second candidate permutation set P k as the initial selection.

[0052] Obtain the connection description of the permutation q through the GetConn(q) function and add these connections to the first candidate permutation G k as the starting point for constructing the topology corresponding to the k-th AllReduce group.

[0053] Calculate the ratio x of several sequences, and this ratio x is used to select the next best permutation.

[0054] Determine the new candidate permutation q' by multiplying the previously selected permutation q by the ratio x, so as to generate candidate permutations based on geometric sequences.

[0055] Select the permutation closest to q' from the set P of second candidate permutations k such that the distance between the added permutation and the existing topology is minimized.

[0056] Obtain the connection description of the permutation q' through the GetConn(q) function and add it to the set G of first candidate permutations k .

[0057] Update the current permutation q to the permutation q', complete one iteration, and perform the iteration d k -1 times. Each time, select a new permutation and add it to the set G of first candidate permutations k .

[0058] Return the finally constructed set G of first candidate permutations k as the sub-topology of the k-th AllReduce group, and merge the topologies corresponding to each AllReduce group to obtain the AllReduce sub-topology, denoted as G AllReduce ={G1, G2,,,G k ,,, G K}, where K is the number of AllReduce groups.

[0059] Step S203, construct the MP sub-topology according to the transmission requirements of the MP traffic and the degree.

[0060] Initialize an empty set as the set G of third candidate permutations MP ={};

[0061] According to the transmission requirements of the MP traffic, perform maximum weight matching through the Blossom Maximum Weight Matching algorithm (Edmonds maximum weight matching algorithm), and add the matched edges to the set G of third candidate permutations MP so that the number of hops passed by the data transmission path in the parallel of the distributed large model is minimized;

[0062] Iteratively optimize the set d of third candidate permutations MP -1 times to obtain the MP sub-topology.

[0063] Step S204, combine the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; and configure optical switches according to the topology graph to implement the physical topology (as Figure 4 shown).

[0064] Furthermore, in this example, the Cion-change idea is also used to calculate the routing rules for the AllReduce topology, and the shortest path algorithm is used to calculate the routing rules for MP. The routes of these two parts are merged to form the final route R

[0065] Step S3: Monitor the traffic changes of all servers during the distributed large model training. When the traffic changes, obtain all distributed large model training task arrays, link arrays, and candidate placement location arrays; where the candidate placement location represents placing a certain task on a certain server; traverse each candidate placement location. For each candidate placement location, construct a corresponding affinity graph according to the topology graph, calculate the compatibility scores of all links in the affinity graph under the condition that the bandwidth requirement satisfies the link capacity constraint, and use the candidate placement location with the highest compatibility score as the best placement location; calculate the latency of all connected subgraphs in the affinity graph corresponding to the best placement location, and use the sum of the latencies of all connected subgraphs as the total latency, with the minimum total latency as the optimization goal; perform distributed large model training on the physical topology according to the best placement location and the total latency.

[0066] Specifically, as Figure 3 shown, the step S3 specifically includes the following sub-steps:

[0067] Step S301: When the traffic changes, obtain all distributed large model training task arrays Jobs, link arrays Links, and candidate placement location arrays Candidates.

[0068] Step S302: Traverse the candidate placement locations. For each candidate placement location c, construct a corresponding affinity graph G c ={U c ,V c ,E c}, as Figure 5 shown, U c represents the task nodes, V c represents the link nodes and E c represents the edge set. For each task j and link l, check whether task j shares a link with other tasks. If link l carries multiple tasks, add these tasks to the node set V c , if task j is passing through link l, create an edge. If there is a loop in the affinity graph G c , remove this candidate location.

[0069] Step S303: For all links, calculate the compatibility scores of tasks passing through the links through the compatibility optimization formula. The compatibility optimization formula is as follows:

[0070]

[0071] Among them, represents the set of jobs competing for a certain link ; represents the bandwidth requirement of the total work task (determined according to routing R) at the rotation angle of ; represents the total capacity of the link bandwidth, represents a set of discrete angles , whose range is [0, 2π], represents the bandwidth requirement at a certain angle α on the unified circle, represents the number of iterations of the job on its unified circle, represents the job on the link rotation angle (in radians), represents the compatibility score of the shared job link .

[0072] Step S304, sort the candidate positions. For all links, sort the candidate placement positions according to the compatibility score, and select the candidate position with the highest score as the best placement position for the task. Obtain the affinity graph G = {U, V, E} corresponding to the best placement position, and traverse each connected subgraph H = {U H , V H , E H} by breadth-first search. For each subgraph, initialize an empty time_shift H to store the time shift in the subgraph.

[0073] Step S305, calculate the unique time shift. Find the edges of each task and link and in the subgraph. Calculate the final time shift and according to the weights of the edges , and use modulo operation to ensure that the time shift does not exceed the range of the iteration time . Add the calculated time shift to , and then add the time shift of each subgraph to . With the best placement combination of tasks and links, and the time offset of tasks, the total delay is minimized as the optimization goal.

[0074] Step S4, when the bandwidth requirement does not meet the link capacity constraint condition, reconstruct the AllReduce sub-topology and MP sub-topology according to the current traffic change, update the topology graph; and reconfigure the optical switch according to the updated topology graph to implement the physical topology;

[0075] Update according to the optimal placement location and the total delay, and continue to perform distributed large model training on the physical topology according to the updated optimal placement location and the total delay.

[0076] Furthermore, an embodiment of the present invention provides a communication scheduling system for distributed large model training, and the system includes:

[0077] A topology reconstruction component, configured to obtain the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of the AllReduce traffic, and the transmission traffic demand of the MP traffic; divide the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission demands of the AllReduce traffic and the MP traffic; construct an AllReduce sub-topology and an MP sub-topology respectively according to the transmission demands and degrees of the AllReduce traffic and the MP traffic; combine the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; configure an optical switch according to the topology graph to implement a physical topology;

[0078] A traffic awareness component, configured to monitor the traffic changes of all servers during the distributed large model training process. When the traffic changes, obtain all distributed large model training task arrays, link arrays, and candidate placement location arrays; where the candidate placement location represents placing a certain task on a certain server; traverse each candidate placement location. For each candidate placement location, construct a corresponding affinity graph according to the topology graph, calculate the compatibility scores of all links in the affinity graph under the condition that the bandwidth demand satisfies the link capacity constraint, and use the candidate placement location with the highest compatibility score as the optimal placement location; calculate the delays of all connected sub-graphs in the affinity graph corresponding to the optimal placement location, and use the sum of the delays of all connected sub-graphs as the total delay, with the minimum total delay as the optimization goal; perform distributed large model training on the physical topology according to the optimal placement location and the total delay.

[0079] Regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0080] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions in the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0081] Correspondingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the communication scheduling method for distributed large model training as described above. As Figure 6 shown, it is a hardware structure diagram of any device with data processing capabilities where the communication scheduling method for distributed large model training provided by the embodiment of the present invention is located. In addition to Figure 6 the processors, memory, and network interfaces shown, any device with data processing capabilities where the device in the embodiment is located usually includes other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated here.

[0082] Correspondingly, this application also provides a computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the communication scheduling method for distributed large model training as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Further, the computer-readable storage medium can also include both the internal storage unit of any device with data processing capabilities and the external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store the data that has been output or will be output.

[0083] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0084] The above embodiments are only used to illustrate the design concept and features of the present invention, and the purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made according to the principles and design concepts disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A communication scheduling method for distributed large model training, characterized in that The method includes: Obtaining the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of AllReduce traffic, and the transmission traffic demand of MP traffic; Dividing the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission demands of AllReduce traffic and MP traffic; constructing the AllReduce sub-topology and the MP sub-topology respectively according to the transmission demands of AllReduce traffic and MP traffic and the degrees; combining the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; configuring an optical switch according to the topology graph to implement a physical topology; Monitoring the traffic changes of all servers during the distributed large model training process, and when the traffic changes, obtaining all distributed large model training task arrays, link arrays, and candidate placement location arrays; where the candidate placement location represents placing a certain task on a certain server; Traversing each candidate placement location, for each candidate placement location, constructing a corresponding affinity graph according to the topology graph, calculating the compatibility scores of all links in the affinity graph under the condition that the bandwidth demand satisfies the link capacity constraint, and taking the candidate placement location with the highest compatibility score as the best placement location; calculating the latency of all connected sub-graphs in the affinity graph corresponding to the best placement location, and taking the sum of the latencies of all connected sub-graphs as the total latency, with the goal of minimizing the total latency; performing distributed large model training on the physical topology according to the best placement location and the total latency; The method further includes: When the bandwidth demand does not satisfy the link capacity constraint condition, reconstructing the AllReduce sub-topology and the MP sub-topology according to the current traffic change, updating the topology graph; and reconfiguring the optical switch according to the updated topology graph to implement the physical topology; Updating according to the best placement location and the total latency, and continuing to perform distributed large model training on the physical topology according to the updated best placement location and the total latency.

2. The communication scheduling method for distributed large model training according to claim 1, wherein The process of obtaining the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of AllReduce traffic, and the transmission traffic demand of MP traffic includes: Obtaining the task information, load pattern, and parallelization strategy of the distributed large model training task; where the task information of the distributed large model training task includes the task type, computing demand, and data scale; the load pattern of the distributed large model training task includes: compute-intensive, data-intensive; the parallelization strategy of the distributed large model training task includes: data parallelism, model parallelism; Obtaining the number of servers for distributed large model training, the degree of each server, the transmission traffic demand of AllReduce traffic, and the transmission traffic demand of MP traffic according to the task information, load pattern, and parallelization strategy of the distributed large model training task.

3. The communication scheduling method for distributed large model training according to claim 1, characterized in that The process of constructing the AllReduce sub-topology includes: For each AllReduce group, allocating the degree corresponding to each AllReduce group according to the traffic corresponding to each AllReduce group; Take the server as a node and an empty set as the initialized first candidate permutation set; calculate the Euler's totient function of the degree corresponding to each server, and the result output by the Euler's totient function represents different connection methods between servers; fill the result output by the Euler's totient function into the second candidate permutation set. In response to the number of servers, the degree corresponding to each AllReduce group, and the second candidate permutation set corresponding to each AllReduce group, construct the first candidate permutation corresponding to each AllReduce group, and iteratively optimize the first candidate permutation through the method of choosing permutations and combinations to obtain the topology corresponding to each AllReduce group. Merge the topologies corresponding to each AllReduce group to obtain the AllReduce sub-topology.

4. A communication scheduling method for distributed large model training according to claim 1, characterized in that The process of constructing the MP sub-topology includes: Initialize an empty set as the third candidate permutation set. According to the transmission requirements of the MP traffic, perform maximum weight matching through the Edmonds maximum weight matching algorithm, and add the matched edges to the third candidate permutation set to minimize the number of hops passed by the data transmission path in the parallel of the distributed large model. Iteratively optimize the third candidate permutation set to obtain the MP sub-topology.

5. The communication scheduling method for distributed large model training according to claim 1, wherein The process of calculating the compatibility score of all links in the affinity graph under the condition that the bandwidth requirement satisfies the link capacity constraint includes: When the bandwidth requirements of all distributed large model training tasks at the rotation angle of α are greater than the total capacity of the current link bandwidth, the compatibility score of the current link is the difference between the bandwidth requirements of all distributed large model training tasks at the rotation angle of α and the total capacity of the current link bandwidth; otherwise, the compatibility score of the current link is 0. Among them, the link capacity constraint is: the bandwidth requirements of all distributed large model training tasks at each angle α are less than or equal to the bandwidth requirements of all distributed large model training tasks at the rotation angle of α, and the rotation angle of the j-th job on the current link is greater than or equal to 0 and less than or equal to the ratio of 2π to the number of iterations of the j-th job on its unified circle.

6. A communication scheduling system for distributed large model training, characterized in that The system includes: A topology reconstruction component, configured to obtain the number of servers for distributed large model training, the degree of each server, the transmission traffic requirements of the AllReduce traffic, and the transmission traffic requirements of the MP traffic; divide the degree of each server into the degrees of the AllReduce traffic sub-topology and the MP traffic sub-topology according to the ratio of the transmission requirements of the AllReduce traffic and the MP traffic; construct the AllReduce sub-topology and the MP sub-topology respectively according to the transmission requirements and degrees of the AllReduce traffic and the MP traffic; combine the AllReduce sub-topology and the MP sub-topology to obtain a topology graph; configure an optical switch according to the topology graph to implement a physical topology. A traffic awareness component is used to monitor the traffic changes of all servers during the training process of a distributed large model. When the traffic changes, it obtains all arrays of distributed large model training tasks, link arrays, and candidate placement location arrays; where a candidate placement location represents placing a certain task on a certain server; it traverses each candidate placement location, and for each candidate placement location, constructs a corresponding affinity graph according to the topology graph, calculates the compatibility scores of all links in the affinity graph under the condition that the bandwidth requirement meets the link capacity constraint, and takes the candidate placement location with the highest compatibility score as the optimal placement location; calculates the latency of all connected subgraphs in the affinity graph corresponding to the optimal placement location, and takes the sum of the latencies of all connected subgraphs as the total latency, with the goal of minimizing the total latency; performs distributed large model training on the physical topology according to the optimal placement location and the total latency; when the bandwidth requirement does not meet the link capacity constraint condition, reconstructs the AllReduce sub-topology and the MP sub-topology according to the current traffic change, updates the topology graph; and reconfigures the optical switch according to the updated topology graph to implement the physical topology; updates according to the optimal placement location and the total latency, and continues to perform distributed large model training on the physical topology according to the updated optimal placement location and the total latency.

7. An electronic device, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the communication scheduling method for distributed large model training according to any one of claims 1-5 above.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the communication scheduling method for distributed large model training according to any one of claims 1-5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the communication scheduling method for distributed large model training according to any one of claims 1-5.

Images