A method and apparatus for topology selection of reduction operation, and a medium

Abstract

The application discloses a topology selection method and device for reduction operation, equipment and medium, comprising: obtaining link information corresponding to each accelerator in a distributed training task, determining the link type between each accelerator and other accelerators according to the link information, detecting the transmission performance and the computing performance corresponding to each link type to obtain transmission performance detection results and computing performance detection results, detecting the control overhead corresponding to each link type to obtain control overhead detection results, and selecting a target topology structure matching the reduction operation link in each accelerator according to the transmission performance detection results, the computing performance detection results, the control overhead detection results and the reduction data volume. The technical scheme of the embodiment of the application can select the optimal topology structure for the reduction operation link in the distributed training task, reduce the time consumption of the distributed training process and the maintenance cost of hardware.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, device and medium for topology selection in reduction operations. Background Technology

[0002] With the development of artificial intelligence (AI) technology, the growth rate of sample data in training models far exceeds the growth rate of chip computing power and storage capacity. A single accelerator card cannot efficiently process large amounts of data, nor can it independently deploy large models. Therefore, distributed computing frameworks have become an inevitable choice for model training.

[0003] In distributed training tasks, multiple accelerator cards can be used simultaneously to complete the training process, and each accelerator card includes an allreduce operation. The processing performance of the allreduce operation is particularly important for the speedup and training efficiency of distributed training. Currently, the allreduce operation typically operates on known topologies, such as ring topologies, 2D ring topologies, and binary tree topologies.

[0004] However, each topology has its own scope of application and limitations. For a specific topology, reduction operations are only high-performance under suitable hardware connections and data volumes. For other hardware connections and data volumes, reducing operations using that topology are low-performance or even unusable. Since existing reduction operations typically run within a specific topology, mismatches between the topology and the corresponding hardware connections and data volumes can easily lead to lengthy distributed training processes and low training efficiency. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for topology selection in reduction operations, which can select the optimal topology for the reduction operation stage in a distributed training task, thereby reducing the time consumption of the distributed training process and the hardware maintenance cost.

[0006] In a first aspect, embodiments of the present invention provide a topology selection method for reduction operations, the method comprising:

[0007] Obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card.

[0008] The transmission performance and computational performance of each link type are detected, and the detection results of the transmission performance and computational performance of each link type are obtained.

[0009] The control overhead corresponding to each link type is probed to obtain the control overhead probe results for each link type.

[0010] Based on the transmission performance detection results, computation performance detection results, and control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, a target topology that matches the protocol operation stage in each accelerator card is selected.

[0011] Secondly, embodiments of the present invention also provide a topology selection apparatus for reduction operations, the apparatus comprising:

[0012] The link detection module is used to obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card.

[0013] The performance detection module is used to detect the transmission performance and computing performance corresponding to each link type, and obtain the transmission performance detection results and computing performance detection results corresponding to each link type.

[0014] The control overhead detection module is used to detect the control overhead corresponding to each link type and obtain the control overhead detection results for each link type.

[0015] The topology selection module is used to select a target topology that matches the protocol operation stage in each accelerator card based on the transmission performance detection results, computation performance detection results, control overhead detection results corresponding to each link type, and the protocol data volume corresponding to each accelerator card.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors execute the programs to implement the topology selection method for reduction operations provided in any embodiment of the present invention.

[0020] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the topology selection method for reduction operations provided in any embodiment of the present invention.

[0021] Fifthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the topology selection method for reduction operations provided in any embodiment of the present invention.

[0022] The technical solution of this invention obtains the link information corresponding to each accelerator card in a distributed training task, determines the link type between each accelerator card and other accelerator cards based on the link information, probes the transmission performance and computing performance corresponding to each link type, obtains the transmission performance probe results and computing performance probe results for each link type, probes the control overhead corresponding to each link type, obtains the control overhead probe results for each link type, and selects the target topology structure that matches the reduction operation stage in each accelerator card based on the transmission performance probe results, computing performance probe results, control overhead probe results, and the reduction data volume corresponding to each accelerator card. This technique can select the optimal topology structure for the reduction operation stage in a distributed training task, reducing the time consumption of the distributed training process and the hardware maintenance cost.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of the topology selection method for reduction operation provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of the topology selection method for reduction operation provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a flowchart of the topology selection method for reduction operation provided in Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of the topology selection device for reduction operation provided in Embodiment 4 of the present invention;

[0029] Figure 5This is a schematic diagram of the structure of an electronic device that implements the topology selection method for reduction operation according to embodiments of the present invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0031] Example 1

[0032] Figure 1 This is a flowchart of a topology selection method for reduction operations provided in Embodiment 1 of the present invention. This embodiment is applicable to selecting a suitable topology for the reduction operation stage in a distributed training task. This method can be executed by a topology selection device for reduction operations. The topology selection device for reduction operations can be implemented by software and / or hardware, and is generally integrated into an electronic device with data processing capabilities. Specifically, it includes the following steps:

[0033] Step 110: Obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card.

[0034] In this embodiment, the distributed training task can be constructed according to the training requirements of the AI ​​model. Multiple accelerator cards can be used simultaneously to complete the training process in the distributed training task. After obtaining the distributed training task, the link information corresponding to each accelerator card can be obtained, such as the globally unique identifier (UUID), the peer globally unique identifier connected to each port via the Cache Coherent Interconnect Protocol (CCIX), the ID of the Peripheral Component Interconnect Express (PCIe), the PCIetree, the host hash, the Remote Direct Memory Access (RDMA) port, and the number of ordinary network ports.

[0035] In this step, after obtaining the link information corresponding to each accelerator card, the link information can be compared with the link information of known links, and the link type between each accelerator card and other accelerator cards can be determined based on the comparison results.

[0036] In a specific embodiment, if the UUID of a certain accelerator card is consistent with the UUID of the peer accelerator card recorded by this card, it means that the link type between the accelerator card and the peer accelerator card is CCIX.

[0037] Step 120: Probe the transmission performance and computing performance corresponding to each link type, and obtain the transmission performance detection results and computing performance detection results corresponding to each link type.

[0038] In this embodiment, optionally, the transmission performance detection result corresponding to each link type can be calculated based on the transmission time of different data volumes in each link type, and the performance detection result can be calculated accordingly. Assuming that under a certain link type, the transmission time corresponding to data volume S1 is t1, and the transmission time corresponding to data volume S2 is t2, where S2 is greater than S1, then the transmission performance calculation formula corresponding to this link type is:

[0039] B = (S2 – S1) / (t2 – t1)

[0040] Where B represents the transmission performance detection result corresponding to this link type.

[0041] In a specific embodiment, assuming the link type between the current accelerator card and another accelerator card is CCIX, the sender and receiver can be determined between the current accelerator card and other accelerator cards, and then steps 121-123 can be executed:

[0042] Step 121: Send a small amount of data S1 through the sending end and receive a small amount of data S1 through the receiving end. The sending time of the sending end and the receiving time of the receiving end are timed separately, and the larger time t1 is selected.

[0043] Step 122: Send a large amount of data S2 through the sending end and receive a large amount of data S2 through the receiving end. The sending time of the sending end and the receiving time of the receiving end are timed separately, and the larger time t2 is selected.

[0044] Step 123: Calculate the transmission performance detection result corresponding to link type CCIX according to the above transmission performance calculation formula, then return to step 121 to obtain multiple transmission performance detection results corresponding to link type CCIX, and take the average value of multiple transmission performance detection results to obtain the average transmission performance detection result corresponding to link type CCIX.

[0045] Finally, using the same method described above, we can calculate the transmission performance detection results for other link types and obtain data such as CCIX bandwidth (Bc), RDMA bandwidth (Br), PCIe bandwidth (Bp), shared memory bandwidth (Bm), and socket bandwidth (Bs) in existing distributed training tasks.

[0046] In this embodiment, the reduction operation includes a reduction operator. The performance detection method of the reduction operator is similar to that of the transmission performance calculation formula, and the performance detection result can be obtained through the following steps:

[0047] Step 124: Send a small data size S1 (e.g., 1 byte) of reduction operator to each accelerator card, and record the total time t1 from sending the reduction operator to its completion.

[0048] Step 125: Send a reduction operator with a large data volume S2 (e.g., 128M bytes) to each accelerator card, and record the total time t2 from sending the reduction operator to its completion.

[0049] Step 126: Calculate the computational performance detection result R2 for each link type according to the above transmission performance calculation formula, where: R2 = (S2 – S1) / (t2 – t1). Then return to step 124 to obtain multiple computational performance detection results for each link type, and take the average value of the multiple computational performance detection results to obtain the average computational performance detection result for each link type.

[0050] Step 130: Probe the control overhead corresponding to each link type to obtain the control overhead probe results for each link type.

[0051] In this embodiment, control overhead refers to the time required from the hardware issuing an instruction to its actual execution during the operation of the accelerator card. Specifically, it includes synchronization overhead, transmission control overhead, and computation control overhead. Optionally, the transmission time, synchronization time, and computation time for a specific amount of data can be calculated separately for each link type, and the calculation results can be used as the control overhead detection results corresponding to the link type.

[0052] Step 140: Based on the transmission performance detection results, computation performance detection results, and control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, select a target topology that matches the protocol operation stage in each accelerator card.

[0053] In practical applications, each topology has a certain range of applicability and usage limitations. For a specific topology, reduction operations are only high-performance under suitable hardware connections and data volumes. For other hardware connections and data volumes, reducing operations using that topology are low-performance or even unusable.

[0054] In this embodiment, optionally, before obtaining the link information corresponding to each accelerator card in the distributed training task, the applicable reduction data volume and link environment (including link transmission performance, computing performance, control overhead, and cluster size) for each topology can be determined. For example, a ring topology is suitable for reduction operations in distributed training tasks with a small cluster size and a small reduction data volume; a 2D ring topology is suitable for reduction operations in tasks with a large cluster size and strong link transmission performance; and a binary tree topology is suitable for reduction operations in tasks with a large cluster size and a moderate or small reduction data volume.

[0055] In this step, after obtaining the transmission performance detection results, computation performance detection results, control overhead detection results, and the protocol data volume corresponding to each link type, the target topology that matches the protocol operation stage can be selected from multiple topologies based on the pre-determined protocol data volume applicable to each topology and the link environment.

[0056] The technical solution of this invention obtains the link information corresponding to each accelerator card in a distributed training task, determines the link type between each accelerator card and other accelerator cards based on the link information, probes the transmission performance and computing performance corresponding to each link type, obtains the transmission performance probe results and computing performance probe results for each link type, probes the control overhead corresponding to each link type, obtains the control overhead probe results for each link type, and selects the target topology structure that matches the reduction operation stage in each accelerator card based on the transmission performance probe results, computing performance probe results, control overhead probe results, and the reduction data volume corresponding to each accelerator card. This technique can select the optimal topology structure for the reduction operation stage in a distributed training task, reducing the time consumption of the distributed training process and the hardware maintenance cost.

[0057] Example 2

[0058] This embodiment is a further refinement of the above embodiments. The same or corresponding terms as those in the above embodiments will not be repeated in this embodiment. Figure 2This is a flowchart of a topology selection method for reduction operation provided in Embodiment 2. In this embodiment, the technical solution can be combined with one or more methods from the above embodiments, such as... Figure 2 As shown, the method provided in this embodiment may further include:

[0059] Step 210: Obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card.

[0060] Step 220: Probe the transmission performance and computing performance corresponding to each link type, and obtain the transmission performance detection results and computing performance detection results corresponding to each link type.

[0061] Step 230: Probe the control overhead corresponding to each link type to obtain the control overhead probe results for each link type.

[0062] Step 240: Input the transmission performance detection results, computation performance detection results, control overhead detection results corresponding to each link type, and the protocol data volume corresponding to each accelerator card into the pre-built protocol computation time model.

[0063] In one embodiment of this example, before obtaining the link information corresponding to each accelerator card in the distributed training task, the method further includes: establishing reduction time calculation formulas corresponding to multiple topologies respectively, and constructing the reduction operation time model according to each of the reduction time calculation formulas.

[0064] In a specific embodiment, taking a ring topology as an example, assuming the reduction data volume is S, the number of AI accelerator cards is Nc, the bandwidth corresponding to the link type is Bx, the control latency corresponding to the link type is Lx, the equivalent computational performance of two numbers performing the reduction operator is R2, and the control latency of the reduction operator is LR, then the reduction time calculation formula for this topology can be:

[0065] T ring =2S*(Nc-1) / Nc / Bx+2*(Nc-1)*Lx+S*((Nc-1) / Nc / R2+((Nc-1)*LR

[0066] ≈2S / Bx+S / R2+(Nc-1)(2Lx+LR)

[0067] In another specific embodiment, taking a two-dimensional ring topology (2d ring) as an example, assuming the horizontal ring link type is X, the bandwidth is Bx, the number of accelerator cards in the horizontal ring is M, the vertical ring data type is Y, the vertical ring bandwidth is By, and the number of accelerator cards in the vertical ring is N, then the corresponding protocol latency calculation formula for this topology can be:

[0068] T 2D-ring =2S / Bx+S / R2+(M-1)(2Lx+LR)+2S / M / By+S / M / R2+(N-1)(2Ly+LR)

[0069] In this embodiment, after establishing reduction time calculation formulas corresponding to multiple topologies, these formulas can be encapsulated to obtain a reduction operation time model.

[0070] Step 250: Using the reduction operation time model, determine the reduction time for each topology among the preset multiple topologies for each accelerator card, and select the target topology that matches the reduction operation stage in each accelerator card based on the reduction time.

[0071] In this embodiment, optionally, the transmission performance detection results, computation performance detection results, control overhead detection results, and protocol data volume corresponding to each link type are input into the protocol operation time consumption model. The protocol operation time consumption model can substitute the transmission performance detection results, computation performance detection results, control overhead detection results, and protocol data volume into the time consumption calculation formula corresponding to each topology, and select the topology with the smaller time consumption as the target topology.

[0072] In a preferred embodiment, the reduction operation time model is used to determine the reduction time for each accelerator card among a preset plurality of topologies, and a target topology matching the reduction operation stage in each accelerator card is selected based on the reduction time, including:

[0073] Step 251: Using the reduction time calculation model, calculate the target reduction time for each topology for each accelerator card under different target data volumes;

[0074] In this embodiment, the target data volume can be a typical data volume commonly used in the reduction operation process, such as 1MB, 10MB, 100M, 1000MB, etc. The target quantity can be specified by the user or automatically set by the electronic device.

[0075] In this step, specifically, the target reduction time for each topology for the accelerator card can be calculated using the following steps:

[0076] Step 2511: Substitute the first target data volume into the reduction time calculation formula of the ring topology in the reduction time calculation model to calculate the reduction time t1;

[0077] Step 2512: Substitute the first target data volume into the reduction time calculation formula of the binary tree topology to calculate the reduction time t2;

[0078] Step 2513: Substitute the first target data volume into the reduction time calculation formula of the two-dimensional ring topology (2d ring) to calculate the reduction time t3; then return to step 2511 to calculate the reduction time of other target data volumes under different topologies.

[0079] Step 252: Based on the reduction time of each target, select the standard topology structure corresponding to each target data volume from multiple topologies;

[0080] In this embodiment, the reduction time of each target data volume under multiple topologies can be compared, and the topology with the shortest time can be used as the standard topology corresponding to the target data volume.

[0081] Step 253: Using the reduction operation time model, determine the reduction time for each accelerator card in the multiple standard topologies, and select the target topology that matches the reduction operation stage in each accelerator card from the multiple standard topologies based on the reduction time.

[0082] In this embodiment, since the amount of reduction data changes dynamically during distributed training, the reduction operation time model can be used to determine the reduction time corresponding to the current reduction data amount for each topology among multiple standard topologies based on the real-time changing amount of reduction data. Finally, the topology with the shortest time is selected as the target topology that matches the current reduction operation stage.

[0083] The advantage of this setup is that it allows for flexible selection of the optimal topology for the reduction operation stage based on changes in the amount of reduction data, thereby reducing the time consumption of the distributed training process and the hardware maintenance costs.

[0084] The technical solution of this invention obtains the link information corresponding to each accelerator card in a distributed training task. Based on the link information, it determines the link type between each accelerator card and other accelerator cards. It then probes the transmission performance and computation performance corresponding to each link type, obtaining transmission performance probe results and computation performance probe results. It also probes the control overhead corresponding to each link type, obtaining control overhead probe results. The transmission performance probe results, computation performance probe results, control overhead probe results, and reduction data volume are input into a reduction operation time model. This model determines the reduction time for each topology corresponding to each accelerator card, and selects a target topology matching the reduction operation stage in each accelerator card based on the reduction time. This approach allows for the selection of the optimal topology for the reduction operation stage in a distributed training task, reducing the time consumption of the distributed training process and the hardware maintenance cost.

[0085] Example 3

[0086] This embodiment is a further refinement of the above embodiments. The same or corresponding terms as those in the above embodiments will not be repeated in this embodiment. Figure 3 This is a flowchart of a topology selection method for reduction operation provided in Embodiment 3. In this embodiment, the technical solution can be combined with one or more methods from the solutions of the above embodiments, such as... Figure 3 As shown, the method provided in this embodiment may further include:

[0087] Step 310: Obtain the link information corresponding to each accelerator card in the distributed training task.

[0088] Step 320: Sequentially obtain one link from the preset link sequence as a candidate link; the link sequence includes multiple links arranged according to priority.

[0089] In this embodiment, before obtaining the link information corresponding to each accelerator card, a link sequence can be constructed according to a preset link priority. Specifically, in the link sequence, CCIX has a higher priority than PCIe, PCIe has a higher priority than shared memory, shared memory has a higher priority than RDMA, and RDMA has a higher priority than socket.

[0090] Step 330: Based on the link information between the current accelerator card and other accelerator cards, determine whether the link corresponding to the current accelerator card is a candidate link. If yes, proceed to step 340; otherwise, return to step 320 to sequentially obtain one link from the preset link sequence as a candidate link until the link corresponding to the current accelerator card is a candidate link.

[0091] Step 340: Select the link type of the candidate link as the link type between the current accelerator card and other accelerator cards.

[0092] In a specific embodiment, it can be determined in turn whether the UUID of other accelerator cards is consistent with the UUID of the peer accelerator card of the current accelerator card. If they are consistent, it means that the link type between the two is CCIX.

[0093] If they are inconsistent, then check whether the host hash of other accelerator cards is the same as the host hash of the current accelerator card. If they are the same, check whether the PCIe Bus ID of other accelerator cards and the PCIe Bus ID of the current accelerator card are under the same PCIe tree. If they are, it means that the link type between them is PCIe; if they are not, it means that the link type between them is shared memory.

[0094] If the host hash of other accelerator cards is different from the host hash of the current accelerator card, then determine whether the server has an RDMA network card. If an RDMA network card exists, then determine that the link type is RDMA.

[0095] If no RDMA network card exists, then check if a regular network card exists on this server. If it does, then the link type is determined to be socket.

[0096] Step 350: Probe the transmission performance and computing performance corresponding to each link type to obtain the transmission performance detection results and computing performance detection results corresponding to each link type.

[0097] Step 360: Probe the transmission control overhead and calculation control overhead corresponding to each link type to obtain the transmission control overhead detection results and calculation control overhead detection results corresponding to each link type.

[0098] In this embodiment, transmission control overhead can be considered as the control and synchronization time for transmitting a very small amount of data (e.g., 1 byte). Specifically, the transmission control overhead detection result can be determined by performing the following steps:

[0099] Step 361: Select a link type as the current processing type, and then start timing according to the start time t1;

[0100] Step 362: Continuously send, receive, and synchronize N very small data volumes in the acceleration card; where N can be set to 100, and the specific value can be preset according to the actual situation. This embodiment does not limit this.

[0101] Step 363: After N data operations are completed, record the end time t2. The transmission control overhead detection result corresponding to the link type can be calculated using the following formula:

[0102] Lc=(t2–t1) / N

[0103] Then return to step 361 to calculate the transmission control overhead detection results for all other link types.

[0104] In this embodiment, similar to the transmission control overhead, the reduction operator control overhead can be considered as the control time for a reduction operation on a very small amount of data (e.g., 1 byte). Specifically, the result of calculating the control overhead can be determined by performing the following steps:

[0105] Step 364: Time the start time t1;

[0106] Step 365: Continuously send N reduction operations with extremely small data volumes to the accelerator card;

[0107] Step 366: After N data operations are completed, record the end time t2. The result of the control overhead detection can be calculated using the following formula:

[0108] LR=(t2–t1) / N

[0109] Step 370: Based on the transmission performance detection results, computation performance detection results, transmission control overhead detection results, computation control overhead detection results, and the protocol data volume corresponding to each link type, select the target topology that matches the protocol operation stage in each accelerator card.

[0110] The technical solution of this invention obtains the link information corresponding to each accelerator card in a distributed training task, sequentially obtains one link from the link sequence as a candidate link, and determines whether the link corresponding to the current accelerator card belongs to the candidate link based on the link information between the current accelerator card and other accelerator cards. If so, the link type of the candidate link is used as the link type between the current accelerator card and other accelerator cards. The transmission performance and computing performance corresponding to each link type are probed to obtain the transmission performance probe results and computing performance probe results. The transmission control overhead and computing control overhead corresponding to each link type are probed to obtain the transmission control overhead probe results and computing control overhead probe results. Based on the transmission performance probe results, computing performance probe results, transmission control overhead probe results, computing control overhead probe results, and reduction data volume, a target topology matching the reduction operation stage is selected. This technique can select the optimal topology for the reduction operation stage in a distributed training task, reducing the time consumption of the distributed training process and the hardware maintenance cost.

[0111] Example 4

[0112] Figure 4This is a schematic diagram of a topology selection device for reduction operation provided in Embodiment 4 of the present invention, as shown below. Figure 4 As shown, the device includes: a link detection module 410, a performance detection module 420, a control overhead detection module 430, and a topology selection module 440.

[0113] Among them, the link detection module 410 is used to obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card.

[0114] The performance detection module 420 is used to detect the transmission performance and computing performance corresponding to each link type, and obtain the transmission performance detection results and computing performance detection results corresponding to each link type.

[0115] The control overhead detection module 430 is used to detect the control overhead corresponding to each link type and obtain the control overhead detection result corresponding to each link type.

[0116] The topology selection module 440 is used to select a target topology that matches the protocol operation stage in each accelerator card based on the transmission performance detection results, computation performance detection results, control overhead detection results corresponding to each link type, and the protocol data volume corresponding to each accelerator card.

[0117] The technical solution provided by this invention obtains the link information corresponding to each accelerator card in a distributed training task, determines the link type between each accelerator card and other accelerator cards based on the link information, probes the transmission performance and computing performance corresponding to each link type, obtains the transmission performance probe results and computing performance probe results corresponding to each link type, probes the control overhead corresponding to each link type, obtains the control overhead probe results corresponding to each link type, and selects the target topology structure that matches the reduction operation stage in each accelerator card based on the transmission performance probe results, computing performance probe results, control overhead probe results, and the reduction data volume corresponding to each accelerator card. This technical means can select the optimal topology structure for the reduction operation stage in the distributed training task, reducing the time consumption of the distributed training process and the hardware maintenance cost.

[0118] Based on the above embodiments, the device further includes:

[0119] The time consumption model construction module is used to establish reduction time consumption calculation formulas corresponding to multiple topologies respectively, and to construct the reduction operation time consumption model according to each of the reduction time consumption calculation formulas.

[0120] Link detection module 410 includes:

[0121] The link acquisition unit is used to sequentially acquire one link from a preset link sequence as a candidate link; the link sequence includes multiple links arranged according to priority;

[0122] The link determination unit is used to determine whether the link corresponding to the current accelerator card is a candidate link based on the link information between the current accelerator card and other accelerator cards.

[0123] The link type processing unit is used to, when the link corresponding to the current accelerator card is a candidate link, use the link type of the candidate link as the link type between the current accelerator card and other accelerator cards.

[0124] The return execution unit is used when the link corresponding to the current accelerator card is not among the candidate links. In this case, the execution returns to retrieve one link from the preset link sequence as a candidate link until the link corresponding to the current accelerator card is among the candidate links.

[0125] The control overhead detection module 430 includes:

[0126] The overhead detection unit is used to detect the transmission control overhead and computation control overhead corresponding to each link type, and obtain the transmission control overhead detection results and computation control overhead detection results corresponding to each link type.

[0127] Topology selection module 440 includes:

[0128] The model input unit is used to input the transmission performance detection results, computation performance detection results, control overhead detection results, and the amount of protocol data corresponding to each link type into the pre-built protocol computation time model.

[0129] The time consumption determination unit is used to determine the reduction time for each topology for each accelerator card in a preset set of multiple topologies through the reduction operation time model, and select the target topology that matches the reduction operation stage in each accelerator card according to the reduction time.

[0130] The target time calculation unit is used to calculate the target reduction time for each topology for each accelerator card under different target data volumes using the reduction time calculation model.

[0131] A standard topology filtering unit is used to filter the standard topology structure corresponding to each target data volume from multiple topology structures based on the reduction time of each target specification.

[0132] The target topology selection unit is used to determine the reduction time of each topology for each accelerator card among multiple standard topologies through the reduction operation time model, and select the target topology that matches the reduction operation stage in each accelerator card among multiple standard topologies based on the reduction time.

[0133] The result processing unit is used to select a target topology that matches the protocol operation stage in each accelerator card based on the transmission performance detection results, computation performance detection results, transmission control overhead detection results, computation control overhead detection results corresponding to each link type, and the protocol data volume corresponding to each accelerator card.

[0134] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in the embodiments of the present invention can be found in the methods provided in all the foregoing embodiments of the present invention.

[0135] Example 5

[0136] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0137] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0138] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the topology selection method for reduction operations.

[0140] In some embodiments, the topology selection method for reduction operations can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the topology selection method for reduction operations described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the topology selection method for reduction operations by any other suitable means (e.g., by means of firmware).

[0141] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0145] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0146] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0147] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A topology selection method for reduction operations, characterized in that, The method includes: Obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card. The transmission performance and computational performance of each link type are detected, and the detection results of the transmission performance and computational performance of each link type are obtained. The control overhead corresponding to each link type is probed to obtain the control overhead probe results for each link type. Based on the transmission performance detection results, computation performance detection results, and control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, a target topology matching the protocol operation stage in each accelerator card is selected. Before obtaining the link information corresponding to each accelerator card in the distributed training task, the process also includes: Determine the appropriate protocol data volume and link environment for each topology; Based on the transmission performance detection results, computation performance detection results, and control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, a target topology matching the protocol operation stage in each accelerator card is selected, including: Based on the transmission performance detection results, computation performance detection results, control overhead detection results, protocol data volume corresponding to each link type, and the protocol data volume and link environment applicable to each topology in advance, a target topology matching the protocol operation stage is selected from multiple topologies.

2. The method according to claim 1, characterized in that, Based on the transmission performance detection results, computation performance detection results, and control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, a target topology matching the protocol operation stage in each accelerator card is selected, including: The transmission performance detection results, computation performance detection results, control overhead detection results corresponding to each link type, and the amount of protocol data corresponding to each accelerator card are input into the pre-built protocol computation time model. Using the reduction operation time model, the reduction time for each topology corresponding to each accelerator card is determined among multiple preset topologies. Based on the reduction time, a target topology matching the reduction operation stage in each accelerator card is selected.

3. The method according to claim 2, characterized in that, Before obtaining the link information corresponding to each accelerator card in the distributed training task, the following steps are also included: Establish reduction time calculation formulas corresponding to multiple topologies respectively, and construct the reduction operation time model based on each of the reduction time calculation formulas.

4. The method according to claim 2, characterized in that, Using the reduction operation time model, among a preset number of topologies, the reduction time for each accelerator card is determined, and based on the reduction time, a target topology matching the reduction operation stage in each accelerator card is selected, including: Using the reduction time calculation model, the target reduction time for each topology and accelerator card is calculated under different target data volumes. Based on the reduction time of each target, a standard topology corresponding to each target data volume is selected from multiple topologies; Using the reduction operation time model, the reduction time for each accelerator card is determined among the multiple standard topologies. Based on the reduction time, a target topology matching the reduction operation stage in each accelerator card is selected from the multiple standard topologies.

5. The method according to claim 1, characterized in that, Based on the link information corresponding to each accelerator card, determine the link type between each accelerator card and other accelerator cards, including: One link from a preset link sequence is selected as a candidate link in turn; the link sequence includes multiple links arranged according to priority. Based on the link information between the current accelerator card and other accelerator cards, determine whether the link corresponding to the current accelerator card is a candidate link; If so, the link type of the candidate link will be used as the link type between the current accelerator card and other accelerator cards; If not, return to the operation of sequentially obtaining one link from the preset link sequence as a candidate link until the link corresponding to the current accelerator card is a candidate link.

6. The method according to claim 1, characterized in that, The control overhead for each link type is probed, and the control overhead probe results for each link type are obtained, including: The transmission control overhead and computational control overhead corresponding to each link type are detected, and the detection results of transmission control overhead and computational control overhead corresponding to each link type are obtained. Based on the transmission performance detection results, computation performance detection results, and control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, a target topology matching the protocol operation stage in each accelerator card is selected, including: Based on the transmission performance detection results, computation performance detection results, transmission control overhead detection results, and computation control overhead detection results corresponding to each link type, as well as the protocol data volume corresponding to each accelerator card, a target topology that matches the protocol operation stage in each accelerator card is selected.

7. A topology selection device for reduction operations, characterized in that, The device includes: The link detection module is used to obtain the link information corresponding to each accelerator card in the distributed training task, and determine the link type between each accelerator card and other accelerator cards based on the link information corresponding to each accelerator card. The performance detection module is used to detect the transmission performance and computing performance corresponding to each link type, and obtain the transmission performance detection results and computing performance detection results corresponding to each link type. The control overhead detection module is used to detect the control overhead corresponding to each link type and obtain the control overhead detection results for each link type. The topology selection module is used to select a target topology that matches the protocol operation stage in each accelerator card based on the transmission performance detection results, computation performance detection results, control overhead detection results corresponding to each link type, and the protocol data volume corresponding to each accelerator card. The device is also used to determine the amount of protocol data and the link environment applicable to each topology before acquiring the link information corresponding to each accelerator card in the distributed training task. The topology selection module is further configured to select a target topology that matches the protocol operation stage from multiple topologies based on the transmission performance detection results, computation performance detection results, control overhead detection results, protocol data volume corresponding to each accelerator card, and the protocol data volume and link environment applicable to each topology structure as determined in advance.

8. An electronic device, the electronic device comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors execute the programs to implement the topology selection method of reduction operation as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the topology selection method of reduction operation as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the topology selection method for reduction operations according to any one of claims 1-6.

Images