Traffic scheduling method and system in distributed training task scene
By using graph coloring and linear programming to optimize path selection, this study solves the problems of intra-task and inter-task flow conflicts in distributed training scenarios, achieving efficient flow scheduling, reducing flow conflict rate and communication time, and improving training efficiency.
Patent Information
- Application Number
- CN202511838355.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-02-13
AI Technical Summary
Existing technologies cannot effectively solve the problems of intra-task and inter-task flow conflicts in distributed training task scenarios, especially in multi-task shared clusters, where network congestion is severe and there is a lack of awareness of the characteristics of distributed training workloads.
A graph coloring method is used to combine network resource allocation with task communication patterns. By establishing a communication model, defining first and second flow scheduling objectives, solving a linear programming problem, optimizing path selection to avoid flow conflicts, and using a knapsack random rounding algorithm to achieve conflict-free path allocation.
It significantly reduces flow collision rate, shortens communication time, improves end-to-end training time, enhances scheduling efficiency, and reduces reliance on precise timing assumptions.
Smart Images
Figure CN121530918A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of communication, and more particularly relates to a traffic scheduling method and system in a distributed training task scenario. BACKGROUND
[0002] Traditional flow scheduling algorithms such as static hash (SH) and equal cost multi-path (ECMP) and other traditional routing algorithms lack information awareness of distributed training (DT) workloads, and when processing distributed training tasks, they select paths for traffic according to some fixed strategies (such as hash tables). Single-task pre-routing algorithm (GSM) is a method of avoiding flow conflicts by analyzing a pre-obtained flow set. Some other methods for scheduling traffic in a multi-task scenario in a graphics processing unit (GPU) cluster are as follows: CASSINI uses a time offset method to stagger the use of the same link by different tasks to reduce interference between tasks; CRUX alleviates network congestion through priority allocation.
[0003] The existing methods for scheduling traffic in a multi-task scenario in a GPU cluster have the following shortcomings: they cannot solve the problem of both intra-task flow conflicts and inter-task flow conflicts. GSM only focuses on the communication performance of a single task, and in a multi-task shared cluster, network congestion between tasks is more serious. Even some existing methods that handle multi-task communication scheduling cannot use task-specific communication patterns for spatial dimension optimization. For example, the effectiveness of CASSINI is limited by the accurate assumption of the length of the computation-communication cycle, which may have more than 20% inaccuracy in actual scenarios; CRUX is a passive congestion handling mechanism that cannot actively prevent congestion; and traditional routing algorithms such as static hash and equal cost multi-path lack awareness of the characteristics of distributed training workloads and cannot effectively solve flow conflicts from the core switch (spine) to the access switch (leaf).
[0004] In summary, the existing technical solutions either target general data center traffic or only optimize a single distributed training task, and none of them can effectively solve the core problem in a shared multi-task cluster, i.e., how to perform efficient communication scheduling when there are both predictable intra-task communication and unpredictable inter-task communication. SUMMARY
[0005] The main purpose of the present application is to provide a traffic scheduling method and system in a distributed training task scenario to overcome the shortcomings of the prior art.
[0006] To achieve the above application purposes, the present application adopts the following technical solutions: The first aspect of the present application provides a traffic scheduling method in a distributed training task scenario, which is used for a distributed training task between a first device end and a second device end, and the method comprises the following steps: S1, according to the communication requirement of the distributed training task, performing bipartite graph abstraction and graph coloring processing on the first device end, the second device end and the network resources between the two in sequence, so as to establish a communication model; S2, based on the communication model, establishing a traffic scheduling optimization problem by combining a first traffic scheduling target and a second traffic scheduling target; wherein the first traffic scheduling target is to select a path through different core switches for a communication flow with the same source address or target address in any communication stage of a single task, and the second traffic scheduling target is to select a path through different core switches for a communication flow with the same source address or target address between different tasks; S3, solving the traffic scheduling optimization problem to obtain a path set; and S4, mapping the communication flow of the distributed training task to the corresponding path in the path set, so as to realize routing.
[0007] Preferably, the S1 specifically comprises: mapping GPUs occupied by the distributed training task in the first device end and the second device end into vertices of a bipartite graph respectively; and establishing a connection link between the corresponding vertices and allocating a color to the link according to the communication flow required to complete the distributed training task, wherein the colors of different links connected with the same vertex are different.
[0008] Preferably, the first traffic scheduling target comprises: ; ; wherein, represents whether a path through a core switch s is used in the pth communication stage of the task j in the distributed training task from the first device end to the second device end , 0 represents not used, and 1 represents used; represents the number of communication flows from the first device end to the second device end in the pth communication stage of the task j in the distributed training task.
[0009] Preferably, the second traffic scheduling target comprises: ; ; ; wherein, represents the number of tasks using the uplink corresponding to the first device end ; denotes using the second device end the number of corresponding downlink tasks; denotes the pth communication stage of task j in the distributed training task, from the first device end to the second device end whether to use the path flowing through the core switch s, 0 indicates not to use, 1 indicates to use.
[0010] Preferably, the S3 specifically comprises: relaxing the traffic scheduling optimization problem into a linear programming problem to replace the discrete variables to be solved with continuous variables; solving the linear programming problem by using a linear programming solver to obtain the optimal solution of the to-be-solved variables; and determining the path for each communication flow according to the optimal solution to obtain the path set.
[0011] Preferably, the step of determining the path for each communication flow according to the optimal solution specifically comprises: converting the non-integer optimal solution into an integer approximate optimal solution ; putting the optimal solution into a knapsack in order, and selecting the path flowing through the core switch s to forward one of the communication flows with a probability , and setting the approximate optimal solution corresponding to the selected communication flow to 1 to obtain the path set; wherein, denotes the pth communication stage of task j in the distributed training task, from the first device end to the second device end ; and is the sum of the approximate optimal solutions in the knapsack. denotes the pth communication stage of task j in the distributed training task, from the first device end to the second device end whether to use the path flowing through the core switch s.
[0012] Preferably, before the S1 is executed, the method further comprises: obtaining parallelization parameters and deployment parameters of the distributed training task; and decomposing the communication load of each task in the distributed training task into a plurality of communication flows according to the parallelization parameters and the deployment parameters to generate the communication demand.
[0013] The second aspect of the present application provides a traffic scheduling system in a distributed training task scenario, for distributed training tasks between a first device end and a second device end, the system comprising: a processing module configured to perform bipartite graph abstraction and graph coloring processing on the first device end, the second device end and network resources therebetween in sequence according to communication requirements of the distributed training task, to establish a communication model; a traffic scheduling optimization problem establishing module configured to establish a traffic scheduling optimization problem based on the communication model, jointly with a first traffic scheduling target and a second traffic scheduling target; wherein the first traffic scheduling target is to select a path for a communication flow with the same source address or target address in any communication phase of a single task to flow through different core switches, and the second traffic scheduling target is to select a path for a communication flow with the same source address or target address between different tasks to flow through different core switches; a solving module configured to solve the traffic scheduling optimization problem to obtain a path set; and a mapping and routing module configured to map the communication flow of the distributed training task to a corresponding path in the path set to realize routing; the system is configured to implement the traffic scheduling method in the distributed training task scenario as described above.
[0014] The third aspect of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the traffic scheduling method in the distributed training task scenario as described above when executing the program.
[0015] The fourth aspect of the present application provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the traffic scheduling method in the distributed training task scenario as described above.
[0016] Compared with the prior art, the present application has the beneficial effects that: a traffic scheduling method and system in a distributed training task scenario are provided, on the one hand, by the method of graph coloring, network resources are allocated to the distributed training task from a global perspective, and flow conflicts between tasks are fundamentally prevented; on the other hand, a first traffic scheduling target for avoiding intra-task flow conflicts and a second traffic scheduling target for avoiding inter-task flow conflicts are designed, by allocating non-overlapping link aggregation to the tasks, prevention of conflicts is realized in the spatial dimension, and dependence on accurate time assumptions is avoided; furthermore, the communication mode characteristics of the distributed training task are fully utilized, the predictable intra-task communication and the unpredictable inter-task communication are processed separately, and the maximum scheduling efficiency is realized; the present application significantly reduces the flow conflict rate, shortens the communication time, and improves the end-to-end training time. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments described in the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.
[0018] Figure 1 The flow chart of the traffic scheduling method in the distributed training task scenario provided by the embodiment of the present application.
[0019] Figure 2 The principle diagram of the abstract graph coloring problem of link allocation provided by the embodiment of the present application.
[0020] Figure 3 The principle diagram of link allocation and intra-task routing provided by the embodiment of the present application.
[0021] Figure 4 The working process schematic diagram of the traffic scheduling method in the distributed training task scenario provided by the embodiment of the present application.
[0022] Figure 5 The flow conflict rate comparison result schematic diagram under different cluster utilization rates.
[0023] Figure 6 The communication time comparison result schematic diagram in the multi-task scenario.
[0024] Figure 7 The training time comparison result schematic diagram in the multi-task scenario.
[0025] Figure 8 The block diagram of the traffic scheduling system in the distributed training task scenario provided by the embodiment of the present application.
[0026] Figure 9 The schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION
[0027] In view of the deficiencies in the prior art, the present inventors have long studied and practiced to propose the technical solutions of the present application. The technical solutions, implementation processes and principles will be further explained as follows.
[0028] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.
[0029] In addition, in the description of the present application, it should be understood that the terms "upper", "lower", "inner", "outer", "horizontal", "vertical", and the like indicate the orientation or positional relationship shown in the drawings based on the orientation or positional relationship, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.
[0030] In the description of the present application, the description of the terms "one embodiment", "an embodiment", "the embodiment" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.
[0031] Figure 1 The flowchart of the traffic scheduling method in the distributed training task scenario provided by the embodiments of the present application. The traffic scheduling method in the distributed training task scenario is used for distributed training tasks between the first device end and the second device end, please refer to Figure 1 , the method comprises operations S1-S4.
[0032] Operation S1, according to the communication demand of the distributed training task, the network resources between the first device end, the second device end and the network resources are processed in turn by bipartite graph abstraction and graph coloring processing, to establish a communication model.
[0033] In a preferred embodiment, before operation S1 is performed, the method further comprises: obtaining parallelization parameters and deployment parameters of the distributed training task; according to the parallelization parameters and the deployment parameters, the communication load of each task in the distributed training task is decomposed into several communication flows to generate the communication demand. The distributed training task contains multiple training tasks.
[0034] In a preferred embodiment, operation S1 specifically comprises the following sub-operations S11-S12.
[0035] In sub-operation S11, the GPUs occupied by the distributed training task in the first device end and the second device end are respectively mapped to the vertices of the bipartite graph.
[0036] In sub-operation S12, according to the communication flow required to complete the distributed training task, connection links are established between the corresponding vertices and the links are allocated colors, wherein the colors of different links connected with the same vertex are different.
[0037] In operation S2, a traffic scheduling optimization problem is established based on the communication model and in combination with the first traffic scheduling target and the second traffic scheduling target. The first traffic scheduling target is to select paths for the communication flows with the same source address or target address in any communication stage of a single task to pass through different core switches. The second traffic scheduling target is to select paths for the communication flows with the same source address or target address between different tasks to pass through different core switches.
[0038] In a preferred embodiment, the first traffic scheduling target includes: ; ; In a preferred embodiment, the second traffic scheduling target includes: ; ; ; wherein, represents whether the path passing through the core switch s is used in the pth communication stage of the task j in the distributed training task from the first device end to the second device end , 0 represents not used, and 1 represents used; represents the number of communication flows from the first device end to the second device end in the pth communication stage of the task j in the distributed training task; represents the number of tasks using the corresponding uplink of the first device end ; represents the number of tasks using the corresponding downlink of the second device end . In operation S3, the traffic scheduling optimization problem is solved to obtain a path set.
[0039] In a preferred embodiment, operation S3 specifically includes the following sub-operations S31-S33.
[0040] In sub-operation S31, the traffic scheduling optimization problem is relaxed into a linear programming problem to replace the discrete variables to be solved with continuous variables.
[0041] In sub-operation S32, a linear programming solver is used to solve the linear programming problem to obtain an optimal solution of the variables to be solved.
[0042] In sub-operation S33, paths are determined for each communication flow according to the optimal solution to obtain the path set.
[0043] In sub-operation S33, paths are determined for each communication flow according to the optimal solution to obtain the path set.
[0044] In a preferred embodiment, sub-operation S33 specifically includes: converting non-integer optimal solutions... Approximate optimal solution converted to integers ; the optimal solution Place them in the knapsack in order, with probability. Select the path through the core switch s for forwarding. One of the communication flows, and the approximate optimal solution corresponding to the selected communication flow. Setting it to 1 yields a set of paths; where, This represents the p-th communication stage of task j in a distributed training task, starting from the first device. To the second device end The number of communication streams; For the backpack The sum of; This represents the p-th communication stage of task j in a distributed training task, starting from the first device. To the second device end Whether to use the optimal solution of the path flowing through core switch s.
[0045] Operation S4 maps the communication flow of the distributed training task to the corresponding path in the path set to achieve routing.
[0046] The technical solution of the present invention will be further described in detail below with reference to several preferred embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Test methods in the following embodiments that do not specify specific conditions are generally performed under conventional conditions.
[0047] The present invention provides a traffic scheduling method for distributed training tasks, applicable to scenarios with multiple distributed training tasks. Its core idea is to combine network resource allocation with task communication modes to simultaneously resolve intra-task and inter-task flow conflicts. This method is mainly achieved through the following two core technical features.
[0048] (1) Core technical feature 1: Definition of network resource partitioning and optimization problem based on graph coloring.
[0049] This invention treats the links between the core switch (spine) and access switches (leaf) in a data center network (e.g., a Leaf-Spine topology) as allocatable network resources. The multi-task communication scheduling problem is formalized as a graph coloring problem, such as... Figure 2 As shown. The application scenarios of this invention are as follows. Figure 3As shown, different device ends are interconnected through access switches leaf and core switches spine, so as to complete data transmission.
[0050] Each GPU occupied by a distributed training task is abstracted as a vertex of a graph or . For each task, each communication phase i thereof can be represented by a bipartite graph In a communication phase, a communication flow can be represented by an edge of the bipartite graph, such as a communication flow demand between communication phases i, and , in the bipartite graph , there is an edge from to . The communication flow can be selected from a plurality of known paths, and is represented by a color .
[0051] The object of the present application is to assign a link (color) to each communication flow (edge) to ensure the following two objects. Object 1: for any communication phase (i.e. graph ) of any task, no two flows with the same source address or destination address select a path flowing through the same spine, which corresponds to the optimization problem, i.e. each graph is "correctly graph colored". Object 2: for communication phases of any two different tasks, reduce flow conflicts as much as possible, i.e. no two flows with the same source address or destination address select a path flowing through the same spine.
[0052] In order to achieve the above two objects, the present application correspondingly proposes a traffic scheduling optimization problem comprising a first traffic scheduling object and a second traffic scheduling object, which includes graph abstraction of network resources and the above two objects.
[0053] For object 1, i.e. correct graph coloring, it can be represented by the following formula: ; ; For object 2, i.e. reducing flow conflicts between tasks as much as possible, the means adopted by the present application is to reduce the path set used by each task at each phase, or in other words, to reduce the degree of multiplexing of each path to different tasks as much as possible. In order to achieve this object, indicators and are proposed to represent how many tasks use an uplink and how many links use a downlink, respectively. Wherein and can be calculated as follows: 、 ; finally, the least degree of path reuse is expected . .
[0054] (2) Core technical feature 2: a knapsack-based random rounding algorithm and deployment of algorithm results.
[0055] Since the communication flow pattern within a single distributed training task is predictable, after the invention assigns a specific set of links to each task, it will further select a conflict-free path for each communication flow of the task within the set, as shown in Figure 3 . The specific steps are as follows.
[0056] First, generate a flow set: according to the parallelization parameters and deployment specifications of the distributed training task, predict the communication flows within the task and group them into disjoint flow sets, and they are concurrent. Since the flow pattern within the task is determined, this step can be accurately completed, thereby ensuring the efficiency of internal communication.
[0057] Second, route calculation: based on the network resource division and problem definition based on graph coloring proposed above, the invention proposes a knapsack-based random rounding algorithm to solve this problem. Specifically, the algorithm consists of three steps: first, relax the traffic scheduling optimization problem into a linear programming problem, replace the discrete variables 、 and with continuous variables; second, calculate the optimal solution 、 and by a linear programming solver (such as PULP); third, determine the path for each flow according to the optimal solution, convert the non-integer optimal solution into an integer approximate optimal solution , put the optimal solution into the knapsack in order, select a path through the core switch s with a probability to forward one of the communication flows, and set the approximate optimal solution corresponding to the selected communication flow to 1, and finally obtain the path set. Finally, map the flow from the first device end r1 to the second device end r2 in the pth phase of task j to the path in the set.
[0058] Finally, route deployment: the final routing configuration (i.e., the forwarding path of each flow) is deployed to the servers in the corresponding device end, and the servers direct the traffic to the specific access switch port, thereby implementing routing.
[0059] The method aims to solve the bottleneck problem caused by communication overhead and flow conflict in a multi-task distributed training environment, and can be implemented by Figure 4 the architecture shown in the figure. Figure 4 The architecture mainly consists of two key components: a performance analyzer and an algorithm core. The performance analyzer is used to generate a set of concurrent flows for each task; the algorithm core is responsible for calculating the routing strategy of the flows. Figure 4 The workflow of the architecture shown in the figure is divided into the following four steps.
[0060] Step 1, query task information. The performance analyzer obtains detailed information of the tasks from the task scheduler, such as parallelization parameters and deployment specifications.
[0061] Step 2, generate flow set. According to these parallelization parameters and deployment specifications, the communication workload of each task is decomposed into multiple flow sets, each of which represents a group of concurrent flows in a communication phase.
[0062] Step 3, run the algorithm of the present application. Using the generated flow set information and network topology structure, the algorithm performs two main tasks: assigning network links to each task; selecting a conflict-free path for each task on the assigned link. This calculation process is usually performed once at task deployment or redeployment.
[0063] Step 4, deploy routing configuration. Distribute the calculated routing solution to the server at each device end to guide the flow to a specific leaf switch port, ensure correct spine switch forwarding and avoid conflict.
[0064] The embodiments of the present application verify the performance of the method by large-scale simulation and small-scale test bench experiments, and compare it with existing solutions such as SH, GSM and ECMP.
[0065] The comparison results of flow conflict rate under different cluster utilization rates are shown in the figure Figure 5 . Referring to Figure 5 , it can be seen that in the simulation experiment, the method of the present application is significantly better than SH and GSM under two deployment strategies. Specifically, under the "first-available" strategy, the total conflict rate of the method of the present application is 34.6% and 40.7% lower than that of SH and GSM, respectively. Under the "idle rack first" strategy, the total conflict rate of the method of the present application is more significantly reduced, reaching 66.1% and 69.7%, respectively.
[0066] The comparison results of communication time in a multi-task scenario are shown in the figure Figure 6 . Referring to Figure 6In the high-shared cluster scenario of the testbed experiment, the communication time of the method is reduced by 33%, 26% and 19% compared with SH, ECMP and GSM respectively. This directly verifies that the method can effectively prevent inter-task congestion, thereby greatly shortening the most time-consuming communication phase.
[0067] The comparison results of training time in the multi-task scenario are shown in Figure 7 Figure 7 In the high-shared cluster environment, the average iteration time of the method is more than 2% higher than that of the other three benchmark schemes. This proves that the method has unique value in handling multi-task shared resources and serious network congestion environments.
[0068] Combining Figures 5-7 It can be shown that the method can significantly reduce the flow conflict rate and reduce the average communication time, especially in a high-shared cluster environment, its performance is better than the existing solutions such as traditional SH, ECMP and GSM.
[0069] Based on the same inventive concept, the present application also provides a traffic scheduling system in a distributed training task scenario, for distributed training tasks between a first device end and a second device end. Referring to Figure 8 , the traffic scheduling system 800 in the distributed training task scenario includes a processing module 810, a traffic scheduling optimization problem establishing module 820, a solving module 830, and a mapping and routing module 840.
[0070] The processing module 810, for example, performs operation S1 for performing bipartite graph abstraction and graph coloring processing on the first device end, the second device end and the network resources therebetween in turn according to the communication requirements of the distributed training task, to establish a communication model.
[0071] The traffic scheduling optimization problem establishing module 820, for example, performs operation S2 for establishing a traffic scheduling optimization problem based on the communication model, jointly with the first traffic scheduling target and the second traffic scheduling target; wherein the first traffic scheduling target is for a communication flow with the same source address or target address in any communication phase of a single task to select a path through different core switches, and the second traffic scheduling target is for a communication flow with the same source address or target address between different tasks to select a path through different core switches.
[0072] The solving module 830, for example, performs operation S3 for solving the traffic scheduling optimization problem to obtain a path set.
[0073] The mapping and routing module 840, for example, performs operation S4 for mapping the communication flow of the distributed training task to the corresponding path in the path set to realize routing.
[0074] The traffic scheduling system 800 in the distributed training task scenario is configured to implement the traffic scheduling method in the distributed training task scenario according to any one of the embodiments.
[0075] Based on the same inventive concept, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the traffic scheduling method in the distributed training task scenario according to any one of the embodiments when executing the program.
[0076] Figure 9 A more specific hardware structure of an electronic device is shown, which can include a processor 910, a memory 920, an input / output interface 930, a communication interface 940, and a bus 950. The processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are connected to each other through the bus 950 for internal communication.
[0077] The processor 910 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.
[0078] The memory 920 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 920 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 920 and executed by the processor 910.
[0079] The input / output interface 930 is configured to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.
[0080] The communication interface 940 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through a wired manner (for example, a USB, a network cable, etc.) or through a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).
[0081] The bus 950 includes a path for transmitting information between various components (for example, the processor 910, the memory 920, the input / output interface 930, and the communication interface 940) of the device.
[0082] It should be noted that although the above device only shows the processor 910, the memory 920, the input / output interface 930, the communication interface 940, and the bus 950, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary for the implementation of the embodiments of the present specification, and does not have to contain all the components shown in the figure.
[0083] The electronic device of the above embodiment is used to realize the traffic scheduling method in the corresponding distributed training task scenario of any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.
[0084] Based on the same inventive concept, the present application also provides a non-transitory computer readable storage medium, which stores computer instructions for causing the computer to execute the traffic scheduling method in the distributed training task scenario as described in any of the above embodiments.
[0085] The computer readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.
[0086] The computer instructions stored in the storage medium of the above embodiments are used to make the computer execute the traffic scheduling method in the distributed training task scenario as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not described here again.
[0087] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary, and is not intended to imply that the scope (including claims) of the present application is limited to these examples; under the idea of the present application, the above embodiments or technical features among different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present application as described above, which are not provided in details for the sake of brevity.
[0088] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present application (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe an exemplary embodiment of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be practiced without these specific details or with variations of these specific details. Therefore, these descriptions should be considered as illustrative rather than limiting.
[0089] Although the present application has been described in conjunction with the specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g. dynamic RAM (DRAM)) can use the embodiments discussed.
[0090] It should be understood that the above embodiments are only to illustrate the technical concepts and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and it cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the spirit and essence of the present application should be covered within the protection scope of the present application.
Claims
1. A traffic scheduling method for a distributed training task scenario, used for distributed training tasks between a first device and a second device, characterized in that... The methods include: S1. Based on the communication requirements of the distributed training task, the first device, the second device, and the network resources between them are sequentially subjected to bipartite graph abstraction and graph coloring to establish a communication model. S2, Based on the communication model, a traffic scheduling optimization problem is established by combining the first traffic scheduling objective and the second traffic scheduling objective; wherein, the first traffic scheduling objective is to select the path of communication flows with the same source address or destination address through different core switches under any communication stage of a single task, and the second traffic scheduling objective is to select the path of communication flows with the same source address or destination address through different core switches between different tasks. S3, Solve the traffic scheduling optimization problem to obtain the path set; S4, map the communication flow of the distributed training task to the corresponding path in the path set to achieve routing.
2. The traffic scheduling method in a distributed training task scenario according to claim 1, characterized in that, S1 specifically includes: The GPUs occupied by the distributed training tasks in the first and second devices are respectively mapped to the vertices of a bipartite graph; Based on the communication flow required to complete the distributed training task, connection links are established between the corresponding vertices and colors are assigned to the links, wherein different links connected to the same vertex have different colors.
3. The traffic scheduling method in a distributed training task scenario according to claim 1, characterized in that, The first traffic scheduling target includes: ； ； in, This represents the p-th communication stage of task j in the distributed training task, starting from the first device. To the second device end Whether to use the path that flows through core switch s. A value of 0 indicates that it is not used. A value of 1 indicates that it is used; This represents the p-th communication stage of task j in the distributed training task, starting from the first device. To the second device end The number of communication streams.
4. The traffic scheduling method in a distributed training task scenario according to claim 1, characterized in that, The second traffic scheduling objective includes: ； ； ； in, Indicates that the first device has been used. The corresponding number of uplink tasks; Indicates that a second device has been used. The corresponding number of downlink tasks; This represents the p-th communication stage of task j in the distributed training task, starting from the first device. To the second device end Whether to use the path that flows through core switch s. A value of 0 indicates that it is not used. A value of 1 indicates that it is used.
5. The traffic scheduling method in a distributed training task scenario according to claim 1, characterized in that, S3 specifically includes: The traffic scheduling optimization problem is relaxed into a linear programming problem, so that the discrete variables to be solved are replaced with continuous variables; The linear programming problem is solved using a linear programming solver to obtain the optimal solution for the variables to be solved. The optimal solution is used to determine the path for each communication flow, thus obtaining the path set.
6. The traffic scheduling method in a distributed training task scenario according to claim 5, characterized in that, Determining paths for each communication flow based on the optimal solution specifically includes: The optimal solution for non-integers Approximate optimal solution converted to integers ; The optimal solution Place them in the knapsack in order, with probability. Select the path through the core switch s for forwarding. One of the communication flows, and the approximate optimal solution corresponding to the selected communication flow. Setting it to 1 yields the set of paths; in, This represents the p-th communication stage of task j in the distributed training task, starting from the first device. To the second device end The number of communication streams; For the backpack The sum of; This represents the p-th communication stage of task j in the distributed training task, starting from the first device. To the second device end Whether to use the optimal solution of the path flowing through core switch s.
7. The traffic scheduling method in a distributed training task scenario according to claim 1, characterized in that, Before executing S1, the method further includes: Obtain the parallelization parameters and deployment parameters of the distributed training task; Based on the parallelization parameters and the deployment parameters, the communication load of each task in the distributed training task is decomposed into several communication streams to generate the communication requirements.
8. A traffic scheduling system for a distributed training task scenario, used for distributed training tasks between a first device and a second device, characterized in that... The system includes: The processing module is used to perform bipartite graph abstraction and graph coloring processing on the first device, the second device, and the network resources between them in sequence according to the communication requirements of the distributed training task, so as to establish a communication model. A traffic scheduling optimization problem establishment module is used to establish a traffic scheduling optimization problem based on the communication model, in conjunction with a first traffic scheduling objective and a second traffic scheduling objective; wherein, the first traffic scheduling objective is to select the path through different core switches for communication flows with the same source address or destination address at any communication stage of a single task, and the second traffic scheduling objective is to select the path through different core switches for communication flows with the same source address or destination address between different tasks. The solution module is used to solve the traffic scheduling optimization problem and obtain a set of paths; The mapping and routing module is used to map the communication flow of the distributed training task to the corresponding path in the path set to achieve routing. The system is used to implement the traffic scheduling method in the distributed training task scenario as described in any one of claims 1-7.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the traffic scheduling method for distributed training task scenarios as described in any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the traffic scheduling method in the distributed training task scenario as described in any one of claims 1-7.