Distributed training set communication control method and device and medium

Abstract

The invention discloses a set communication control method and device for distributed training and a medium, and the method comprises the steps: reducing data on all calculation nodes in a first cluster to a specified calculation node when the difference of the number of calculation nodes between any two clusters in clusters participating in data reduction is within a preset range, the number of the specified computing nodes is the same as the number of the computing nodes in the second cluster with the minimum number of the computing nodes. Controlling the specified computing node and the computing nodes in the second cluster to perform data reduction; and controlling data synchronization between the specified computing node and other nodes except the specified computing node in the first cluster. Therefore, other clusters except the cluster with the minimum number of computational nodes are reduced once in the cluster, the data are reduced to the specified computational nodes with the number the same as that of the minimum nodes in each cluster, it is guaranteed that the nodes of each cluster are the same during cross-cluster reduction, and it is avoided that some nodes are reduced with multiple nodes at the same time; and the set communication overhead is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and medium for collective communication control of distributed training. Background Art

[0002] With the rapid development of big data and artificial intelligence (AI) models, distributed training technology has been widely used in AI models so that the complex computing tasks of AI models can be completed through the collaboration of multiple computing nodes in a computing cluster. That is, the tasks originally completed on a single computing node are decomposed into multiple computing nodes to collaboratively complete complex computing tasks on an AI scale.

[0003] Collective communication refers to a specific form of communication between a group of processes or computing nodes. In the distributed training of AI models, optimizing collective communication performance is the main technical means to improve the efficiency of distributed training. Improving the efficiency of distributed training is crucial to ensure that each computing node in the cluster can fully exert its computing performance. At present, collective communications are mostly built based on homogeneous networks. When optimizing the collective communication performance of homogeneous networks, data is evenly divided between computing nodes, and the data is reduced to each computing node through collective communication semantics to achieve optimal communication bandwidth, thereby improving distributed training efficiency.

[0004] However, in order to meet the explosive growth of intelligent computing power requirements of AI models, various AI chips are constantly updated, resulting in mixed deployment of different AI chips in AI models. Different AI chips have different computing power, which will lead to different numbers of computing nodes participating in mixed training between different computing clusters. Therefore, when performing data reduction across clusters, due to the asymmetric number of computing nodes between clusters participating in the reduction, some computing nodes will need to perform data reduction with multiple computing nodes at the same time, resulting in network competition, resulting in high collective communication overhead, low cross-cluster reduction efficiency, and reduced efficiency of distributed training.

[0005] It can be seen that how to solve the high collective communication overhead and low cross-cluster reduction efficiency and improve the efficiency of distributed training is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0006] In view of this, one aspect of the present application provides a collective communication control method for distributed training, the method comprising: When the difference in the number of computing nodes between any two clusters in the clusters participating in data reduction is within a preset range, the data distributed on all computing nodes in the first cluster are reduced to the designated computing nodes in the first cluster; wherein the number of the designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes in the clusters participating in data reduction; Controlling the designated computing node to perform data reduction with computing nodes in the second cluster; Control the designated computing node to synchronize data with other nodes in the first cluster except the designated computing node.

[0007] Optionally, reducing the data distributed on all computing nodes in the first cluster to a designated computing node in the first cluster includes: Divide the computing nodes in the first cluster into m node groups; wherein each of the node groups has k computing nodes, and k=floor(n / m), n is the number of computing nodes in the first cluster, and m is the number of computing nodes in the second cluster; Selecting a computing node from each of the node groups as the designated computing node; The data distributed on all computing nodes in the first cluster are reduced to the designated computing node.

[0008] Optionally, selecting a computing node from each of the node groups as the designated computing node includes: Sort the computing nodes in the first cluster according to their device serial numbers from small to large; Based on the sorting result, the k computing nodes are divided into the node groups in sequence in a manner that each node group divides one computing node at a time; The computing node with the smallest device serial number in each of the node groups is used as the designated computing node.

[0009] Optionally, reducing the data distributed on all computing nodes in the first cluster to the designated computing node includes: Determine whether there is a target computing node that has not been grouped in the computing nodes in the first cluster; If not, the data in each of the node groups is reduced to the designated computing nodes in the corresponding group; If so, execute the step of reducing the data in each of the node groups to the designated computing nodes in the corresponding group; and divide the data on the target computing node into m portions of data on average, and send the m portions of data to different designated computing nodes in a one-to-one correspondence manner.

[0010] Optionally, the collective communication control method for distributed training further includes: When the gap is not within the preset range, the computing nodes are sorted in ascending order of number; Based on the ranking results, the first and second clusters are merged into the target cluster; Determine whether the gap between any two clusters in the target cluster and the unmerged clusters is within a preset range; If yes, the target cluster and the unmerged cluster are used as the clusters participating in data reduction; and the step of reducing the data distributed on all computing nodes in the first cluster to the designated computing nodes in the first cluster is entered, and subsequent steps are executed.

[0011] Optionally, if the gap between any two clusters in the target cluster and the unmerged cluster is not within the preset range, the method further includes: Determine whether the difference between the number of computing nodes in the target cluster and the number of computing nodes in a specified cluster exceeds a threshold; wherein the specified cluster is the cluster with the smallest number of computing nodes in the unmerged clusters; If the threshold is exceeded, the step of taking the target cluster and the unmerged cluster as the clusters participating in data reduction is entered, and subsequent steps are performed; If the threshold is not exceeded, determine whether the number of the unmerged clusters is greater than 1; if it is greater, enter the step of sorting the computing nodes in ascending order, and execute subsequent steps; if it is not greater, enter the step of taking the target cluster and the unmerged cluster as the clusters participating in data reduction, and execute subsequent steps.

[0012] Optionally, the difference in the number of computing nodes between clusters participating in data reduction is within a preset range, including: If the result of a specified calculation of the maximum and minimum numbers of computing nodes in the cluster participating in data reduction is not greater than a preset value, the gap is within the preset range; wherein the specified calculation includes a ratio calculation or a difference calculation.

[0013] Another aspect of the present application provides a collective communication control device for distributed training, the device comprising: The cluster internal reduction module is used to reduce the data distributed on all computing nodes in the first cluster to the designated computing nodes in the first cluster when the difference in the number of computing nodes between any two clusters in the clusters participating in the data reduction is within a preset range; wherein the number of the designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes in the clusters participating in the data reduction; A cross-cluster reduction module, used to control the designated computing node and the computing nodes in the second cluster to perform data reduction; The data synchronization module is used to control the designated computing node to synchronize data with other nodes in the first cluster except the designated computing node.

[0014] Another aspect of the present application provides a collective communication control device for distributed training, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the steps of the collective communication control method for distributed training are implemented.

[0015] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the collective communication control method for distributed training when the program is executed by a processor.

[0016] The present application provides a collective communication control method, device and medium for distributed training, which have the following beneficial effects: in distributed training, when cross-cluster data reduction is performed, if the number of computing nodes between clusters is different, the clusters other than the cluster with the least number of computing nodes first perform a data reduction within the cluster, so as to reduce the data to the designated computing nodes whose number is the same as the minimum number of computing nodes in each cluster, thereby ensuring that the number of computing nodes between different clusters is aligned during cross-cluster reduction, that is, the number of computing nodes is the same, avoiding the need for some computing nodes to perform data reduction with multiple computing nodes at the same time, resulting in network contention, reducing the overhead of collective communication, and improving the efficiency of cross-cluster reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a collective communication control method for distributed training provided in an embodiment of the present application; Figure 2 A schematic diagram of the principle of a collective communication control method for distributed training provided in an embodiment of the present application; Figure 3 A flowchart of a collective communication control method for distributed training provided by another embodiment of the present application; Figure 4A schematic diagram of the principle of a collective communication control method for distributed training provided by another embodiment of the present application; Figure 5 A schematic diagram of the principle of a collective communication control method for distributed training provided by another embodiment of the present application; Figure 6 A flowchart of a collective communication control method for distributed training provided by another embodiment of the present application; Figure 7 A schematic diagram of the structure of a collective communication control device for distributed training provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a collective communication control device for distributed training provided in another embodiment of the present application.

[0018] The reference numerals are as follows: 80 is a memory, 81 is a processor, 82 is a display screen, 83 is an input / output interface, 84 is a communication interface, 85 is a power supply, 86 is a communication bus, 801 is a computer program, 802 is an operating system, and 803 is data. DETAILED DESCRIPTION

[0019] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] Figure 1 A flow chart of a collective communication control method for distributed training provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes: S10: When the difference in the number of computing nodes between any two clusters in the clusters participating in the data reduction is within a preset range, the data distributed on all computing nodes in the first cluster are reduced to the designated computing nodes in the first cluster; wherein the number of the designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes in the clusters participating in the data reduction; It is understandable that in the distributed training with mixed deployment of different AI chips, the different AI chips have different computing powers, which leads to different numbers of computing nodes in the clusters involved in data reduction. For example, in a deep learning model of a complex image recognition training task, two types of AI chips are required to complete it together, including chip A and chip B, where the intelligent computing power of chip A is higher than that of chip B. Therefore, 4 chips A are deployed in cluster A and 10 chips B are deployed in cluster B to jointly complete the image recognition task of the deep learning model. Obviously, the number of AI chips in cluster A and cluster B is different, that is, the number of computing nodes is different.

[0022] In distributed training, communication within the same computing node can use high-speed interconnect technologies such as NVLink, and communication between different computing nodes in the same cluster can use high-speed interconnect technologies such as GDR (GPU Direct RDMA) for communication, while communication between different clusters can only use the relatively low-speed RoCE (RDMA over ConvergedEthernet) interconnect technology.

[0023] It can be seen from this that in a hybrid distributed training scenario with heterogeneous communication links, when collective communication optimization is performed to improve the efficiency of distributed training, due to the different numbers of computing nodes between clusters participating in data reduction, some computing nodes may need to perform data reduction with multiple computing nodes at the same time when participating in data reduction, resulting in network competition and affecting the collective communication optimization effect.

[0024] Because, in order to solve the above technical problem, in an optional embodiment, the number of computing nodes between clusters participating in data reduction is aligned, that is, the number of computing nodes between clusters participating in data reduction is the same.

[0025] Specifically, the data distributed on all computing nodes in the first cluster are reduced to the designated computing nodes in the first cluster, wherein the number of designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes among the clusters participating in the data reduction. In other words, the number of computing nodes of all clusters participating in the data reduction is aligned with the cluster with the smallest number of computing nodes. For ease of understanding, an example will be given below.

[0026] For example, the clusters involved in data reduction include cluster A, cluster B, and cluster C, and the number of computing nodes in cluster A is 15, the number of computing nodes in cluster B is 20, and the number of computing nodes in cluster C is 23. Obviously, in this example, the first clusters are cluster B and cluster C, and the second cluster with the smallest number of computing nodes is cluster A, and the designated number of computing nodes should be aligned with the number of computing nodes in cluster A, that is, the designated number of computing nodes is 15.

[0027] When cluster A, cluster B, and cluster C perform cross-cluster data reduction, in order to ensure that the number of computing nodes among the three clusters is the same, cluster B and cluster C are first controlled within their respective clusters to reduce the data distributed on all computing nodes to 15 designated computing nodes.

[0028] S11: Control the designated computing node to perform data reduction with computing nodes in the second cluster; Furthermore, after the number of computing nodes in each cluster is aligned, cross-cluster reduction is performed. Specifically, the designated computing nodes are controlled to perform data reduction with the computing nodes in the second cluster. For example, in the above example, the 15 designated computing nodes on clusters B and C are controlled to perform data reduction between two clusters with the 15 computing nodes in cluster A. That is, the reduction includes the reduction between cluster B and cluster A, cluster B and cluster C, and cluster A and cluster C.

[0029] It is worth noting that if the number of computing nodes between any two clusters participating in data reduction is very different, it will lead to insufficient utilization of communication links and waste of resources. For example, there are 5 computing nodes in cluster A and 50 computing nodes in cluster B. When cross-cluster data reduction is performed, cluster B will first reduce the data to 5 designated computing nodes, and then use these 5 computing nodes to reduce the data with the 5 computing nodes in cluster A.

[0030] Obviously, this will result in many communication links being idle during cross-cluster reduction, and the efficiency will be relatively low during data reduction due to the large amount of data. Therefore, in an optional embodiment, before reducing the data distributed on all computing nodes in the first cluster to the designated computing nodes in the first cluster, it is first determined whether the difference in the number of computing nodes between any two clusters participating in the data reduction is within a preset range.

[0031] That is to say, before a cluster with more computing nodes performs data reduction internally, it first determines whether the number of computing nodes among all clusters participating in the data reduction is relatively uniform, that is, the difference in the number of computing nodes cannot be too large.

[0032] It is worth noting that when determining whether the difference in the number of computing nodes between any two clusters is within a preset range, it can be judged based on the difference in the number of computing nodes between any two clusters, or it can be determined based on the ratio of the computing nodes between any two clusters. It can also be determined by other methods whether the number of computing nodes between any two clusters is relatively uniform, and this application does not limit this.

[0033] Figure 2 A schematic diagram of the principle of a distributed training collective communication control method provided in an embodiment of the present application. In an optional embodiment, if the number of computing nodes between the clusters participating in data reduction is the same, such as Figure 2 As shown in the figure, the number of computing nodes in clusters 1 to cluster N is 4. When performing cross-cluster data reduction, the computing nodes within each cluster evenly divide the data, and each computing node is only responsible for reducing a small part of the data corresponding to itself. Furthermore, data reduction is completed between computing nodes within the cluster through the collection communication semantics of Reduce Scatter, that is, data exchange is completed. Then, cross-cluster reduction is performed to complete the data reduction between clusters.

[0034] S12: Control the designated computing node to synchronize data with other nodes in the first cluster except the designated computing node.

[0035] Furthermore, after completing the cross-cluster data reduction, the designated computing node is controlled through step S12 to synchronize data with other nodes in the first cluster except the designated computing node. For example, in the above example, among the 20 computing nodes in cluster B, after 15 designated computing nodes complete data reduction with clusters A and C, these 15 designated computing nodes need to synchronize data to the other 5 computing nodes that have not performed cross-cluster data reduction. Similarly, cluster C also needs to perform the same data synchronization operation. In cluster A, since all computing nodes participate in the cross-cluster data reduction, there is no need to synchronize data.

[0036] Therefore, in the collective communication control method for distributed training provided in the embodiment of the present application, when cross-cluster data reduction is performed in distributed training, if the number of computing nodes between clusters is different, other clusters except the cluster with the least number of computing nodes first perform data reduction within the cluster, so as to reduce the data to the designated computing nodes whose number is the same as the minimum number of computing nodes in each cluster, thereby ensuring that the number of computing nodes between different clusters is aligned during cross-cluster reduction, that is, the number of computing nodes is the same, avoiding the need for some computing nodes to perform data reduction with multiple computing nodes at the same time, resulting in network contention, reducing the overhead of collective communication, and improving the efficiency of cross-cluster reduction.

[0037] Figure 3 A flow chart of a collective communication control method for distributed training provided by another embodiment of the present application, in an optional embodiment, such as Figure 3 As shown, reducing the data distributed on all computing nodes in the first cluster to a specified computing node in the first cluster includes: S30: Divide the computing nodes in the first cluster into m node groups; wherein each node group has k computing nodes, and k=floor(n / m), n is the number of computing nodes in the first cluster, and m is the number of computing nodes in the second cluster; In an optional embodiment, when selecting the designated computing node in the first cluster, the selection can be performed according to certain rules. Specifically, the computing nodes in the first cluster are first divided according to the number of computing nodes in the second cluster. Therefore, the number of computing nodes in the first cluster and the second cluster is first counted, and the number of computing nodes in the first cluster is recorded as n, and the number of computing nodes in the second cluster is recorded as m. Obviously, n>m.

[0038] Furthermore, the n computing nodes in the first cluster are divided into m node groups, each of which includes k computing nodes. In order to ensure that the number of computing nodes in each node group is the same and that there are an integer number of computing nodes in each node group, k=floor(n / m), that is, the number of computing nodes in each node group is equal to the number of computing nodes in the first cluster divided by the number of computing nodes in the second cluster, rounded down. For example, when n=10 and m=3, k=floor(n / m)=3.

[0039] S31: Select a computing node from each node group as a designated computing node; S32: Reduce the data distributed on all computing nodes in the first cluster to the designated computing node.

[0040] Further, a computing node is selected from each node group as the designated computing node. The selection may be random or based on the device serial number of the computing node, which is not limited in this application.

[0041] It is understandable that by dividing the computing nodes in the first cluster into m node groups and selecting a designated computing node from each node group, m designated computing nodes can be finally obtained. Furthermore, the data distributed on all computing nodes in the first cluster can be reduced to the designated computing nodes selected in this embodiment.

[0042] Therefore, the collective communication control method for distributed training provided in the embodiment of the present application aligns the number of computing nodes between clusters participating in data reduction, that is, aligns with the cluster with the least number of computing nodes. This avoids the need for some computing nodes to reduce with multiple computing nodes at the same time when performing data reduction across clusters, optimizes collective communication, and improves distributed training efficiency.

[0043] As an optional embodiment, selecting a computing node from each node group as a designated computing node includes: Sort the computing nodes in the first cluster according to their device serial numbers from small to large; Based on the sorting results, k computing nodes are divided into node groups in turn, with each node group dividing one computing node at a time; The computing node with the smallest device serial number in each node group is used as the designated computing node.

[0044] It is understandable that different computing nodes have different device serial numbers. Computing nodes with the same device serial number can use one switch to communicate, while computing nodes with different device serial numbers require multiple switches to communicate.

[0045] Therefore, in an optional embodiment, in order to save resources, when selecting a computing node from each node group as the designated computing node, the computing node with the smallest device serial number in each node group is selected as the designated computing node.

[0046] Specifically, the computing nodes in the first cluster are first sorted from small to large according to their device serial numbers. After sorting, the k computing nodes are divided into the node groups one by one in each node group. That is, after sorting from small to large, one computing node is placed into m node groups at a time, and the node group where the currently placed computing node is located is different from the node group where the computing node was placed last time.

[0047] It is understandable that, since the number of computing nodes in the second cluster is the smallest, the maximum value of the device serial number of the computing node is smaller than the maximum value of the device serial number in the first cluster. Therefore, in order to align with the device serial number in the second cluster and save the device resources of the switch, the computing node with the smallest device serial number in each node group is used as the designated computing node. Figure 4 A schematic diagram of the principle of a collective communication control method for distributed training provided in another embodiment of the present application will be illustrated below with examples for ease of understanding.

[0048] For example, Figure 4As shown, the clusters involved in data reduction include cluster A and cluster B, and the number of computing nodes in cluster A is 7, and the number of computing nodes in cluster B is 3. Obviously, in this example, the first cluster is cluster A, and the second cluster with the smallest number of computing nodes is cluster B, and the number of designated computing nodes is 3. The device serial number of each computing node in cluster B is 0 to 2, and the device serial number of each computing node in cluster A is 0 to 6, and the number of designated computing nodes is 3, that is, the number of node groups m=3, and the number of computing nodes in each node group k=2.

[0049] When determining the specified computing node in cluster A, such as Figure 4 As shown, the seven computing nodes in cluster A are divided into three node groups including node group A1, node group A2 and node group A3. The seven computing nodes are sorted from small to large according to their device serial numbers.

[0050] Furthermore, according to the sorting results, the seven computing nodes are divided into three node groups in turn, one computing node is divided at a time, and the node groups divided each time are different, that is, the node groups to which the computing nodes divided twice belong are different. Figure 4 As shown, the computing nodes of device number 0 and device number 3 are in node group A1, the computing nodes of device number 1 and device number 4 are in node group A2, and the computing nodes of device number 2 and device number 5 are in node group A5.

[0051] In order to ensure that the device serial numbers of the designated computing nodes selected from the three node groups are aligned with the device serial numbers in cluster B, the computing nodes with the smallest device serial numbers in each node group are used as designated computing nodes, that is, the computing nodes with device serial number 0 in node group A1, the computing nodes with device serial number 1 in node group A2, and the computing nodes with device serial number 2 in node group A3 are used as designated computing nodes.

[0052] Further, in an optional embodiment, when controlling the designated computing node to perform data reduction with the computing nodes in the second cluster, the computing nodes in the designated computing node whose device serial numbers are the same as the computing nodes in the second cluster are subjected to corresponding data reduction. For example, in the above example, the computing node with device serial number 0 in node group A1 is reduced with the computing node with device serial number 0 in cluster B, and the same is true for other computing nodes.

[0053] Therefore, the collective communication control method for distributed training provided in the embodiment of the present application selects the computing node with the smallest device serial number in the node group as the designated computing node so as to align with the device serial number of the computing node in the second cluster, thereby optimizing collective communication to improve distributed efficiency while saving device resources.

[0054] In an optional embodiment, reducing the data distributed on all computing nodes in the first cluster to the designated computing node includes: Determine whether there is a target computing node that has not been grouped in the computing nodes in the first cluster; If it does not exist, the data in each node group is reduced to the designated computing node in the corresponding group; If it exists, execute the step of reducing the data in each node group to the designated computing node in the corresponding group; and divide the data on the target computing node into m portions evenly, and send the m portions to different designated computing nodes in a one-to-one correspondence manner.

[0055] It is understandable that when grouping the computing nodes in the first cluster, it may be impossible to divide all nodes into node groups, wherein the number of target computing nodes that are not grouped is t=nk*m. Figure 4 In the example shown, after the seven clusters in cluster A are divided into three node groups, there is still one computing node that cannot be divided into a node group.

[0056] At this time, in an optional embodiment, when reducing the data distributed on all computing nodes in the first cluster to a designated computing node, it is necessary to first determine whether there is an ungrouped target computing node in the current first cluster.

[0057] If it exists, in order to ensure that all data in the first cluster participate in the cross-cluster reduction, the data on the target computing node that does not participate in the grouping is divided into m pieces of data, that is, the number of divisions is the same as the number of designated computing nodes. Further, the m pieces of data are sent to different designated computing nodes in a one-to-one correspondence. For ease of understanding, the following will combine Figure 4 Provide explanation.

[0058] like Figure 4 As shown, the target computing node participating in the grouping is the node with device number 6, and the shaded parts in different computing nodes are used to represent the data in the current computing node. For example, the data on the target computing node with device number 6 is evenly divided into 3 parts, including data P61, data P62 and data P63. Further, the data on the target computing node is sent (Pend) to different designated computing nodes in a one-to-one correspondence. That is, data P61 is sent to the computing node with device number 0, data P62 is sent to the computing node with device number 1, and data P63 is sent to the computing node with device number 2.

[0059] Of course, it should be noted that this application does not limit the correspondence between the m pieces of data divided by the target computing node and the designated computing nodes, as long as different data are sent to different designated computing nodes.

[0060] In a specific embodiment, while the target computing node sends data to the designated computing node, the data in each node group is also reduced to the designated computing node in the corresponding group through the collective communication semantics All Reduce.

[0061] For example, Figure 4 As shown, data P3 on the computing node of device number 3 is reduced to the designated computing node of device number 0 in the corresponding group through the collective communication semantics AllReduce, data P3 on the computing node of device number 4 is reduced to the designated computing node of device number 1 in the corresponding group, and data P3 on the computing node of device number 5 is reduced to the designated computing node of device number 2 in the corresponding group.

[0062] It is worth noting that the step of reducing the data in each node group to the designated computing node in the corresponding group and the step of dividing the data on the target computing node into m portions of data on an equal basis and sending the m portions of data to different designated computing nodes in a one-to-one correspondence can be performed simultaneously or in a certain order, and this application does not impose any limitation on this.

[0063] In another optional embodiment, if there are no ungrouped target computing nodes in the computing nodes in the first cluster, the number of computing nodes representing the current first cluster can be evenly divided into m node groups. At this time, the data in each node group can be directly reduced to the designated computing nodes in the corresponding group.

[0064] Figure 5 A schematic diagram of the principle of a collective communication control method for distributed training provided by another embodiment of the present application. Further, in an optional embodiment, after data reduction is completed within the cluster, that is, the data in the node group is reduced to the designated computing node, and the target computing node that has not performed data reduction also sends the data evenly to each designated computing node, such as Figure 4 As shown, data reduction is performed between designated computing nodes in each cluster through the collective communication semantics of ReduceScatter.

[0065] Further, such as Figure 5 As shown in , after cross-cluster data reduction, the designated computing nodes corresponding to different node groups in cluster A perform data reduction through the collective communication semantics of All Gather. Figure 5 As shown, in the same node group, the designated computing node broadcasts data to other computing nodes in the node group through the collective communication semantics of Broadcast.

[0066] In another optional embodiment, when there are target computing nodes that are not grouped in the computing nodes in the first cluster, such as Figure 5 As shown in the figure, after the designated computing node completes the cross-cluster reduction, in addition to reducing data with other designated computing nodes in the node and broadcasting data to other computing nodes in the same node group, the designated computing node also needs to broadcast data to ungrouped target computing nodes through the collective communication semantics of Broadcast, thus completing the cross-cluster data reduction.

[0067] Figure 6 A flow chart of a distributed training collective communication control method provided by another embodiment of the present application is provided as an optional embodiment. Figure 6 As shown, the collective communication control method for distributed training provided by the present application, when the gap is not within a preset range, further includes: S60: sorting the computing nodes in ascending order of number; S61: Based on the ranking result, merge the first cluster and the second cluster into a target cluster; In a specific embodiment, if the number of computing nodes between any two clusters participating in data reduction is very different, it will lead to insufficient utilization of communication links and waste of resources. Therefore, in order to solve this technical problem, in an optional embodiment, when the difference is not within a preset range, the number of computing nodes is sorted in ascending order, and the first cluster and the second cluster are merged into the target cluster, that is, the cluster with the smallest number and the cluster with the second smallest number are merged to obtain the target cluster.

[0068] S62: Determine whether the gap between any two clusters in the target cluster and the unmerged clusters is within a preset range; if yes, proceed to step S63.

[0069] S63: taking the target cluster and the unmerged cluster as clusters participating in data reduction; and proceeding to step S10 in the above embodiment, and executing subsequent steps S11 and S12.

[0070] Further, after merging to obtain the target cluster, determine whether the difference between any two clusters in the target cluster and the unmerged cluster is within a preset range, that is, determine whether the number of computing nodes between the clusters is relatively uniform after the merger. If the difference in the number of computing nodes between the clusters after the merger is within an acceptable range, that is, relatively uniform, cross-cluster data reduction can be performed, and step S63 is executed at this time.

[0071] Specifically, the target cluster and the unmerged clusters are used as new clusters participating in data reduction, and cross-cluster reduction is implemented through steps S10 to S12 in the above embodiment. For ease of understanding, an example will be given below.

[0072] For example, the clusters currently involved in data reduction include cluster 1, cluster 2, and cluster 3, and the number of computing nodes in cluster 1 is 5, the number of computing nodes in cluster 2 is 50, and the number of computing nodes in cluster 3 is 60. Obviously, if clusters 1, cluster 2, and cluster 3 perform data reduction, clusters 2 and cluster 3 need to reduce the data to 5 designated computing nodes within the cluster first, which will seriously cause some communication links to be idle during cross-cluster reduction, and the computing nodes involved in the reduction have a large amount of data, and the reduction efficiency is also very low.

[0073] Therefore, in the technical solution provided in the embodiment of the present application, since the number of computing nodes in the three clusters varies greatly, at this time, after sorting the number of computing nodes in the three clusters, cluster 1 and cluster 2 are merged to obtain the target cluster, and the obtained target cluster includes 55 computing nodes.

[0074] Furthermore, both the target cluster and cluster 3 are used as new clusters participating in data reduction. Obviously, the difference between the number of 55 computing nodes and 60 computing nodes is small, which can avoid more idle links during cross-cluster reduction.

[0075] Therefore, the collective communication control method for distributed training provided in the embodiment of the present application, when the number of computing nodes between clusters participating in data reduction differs greatly, merges the clusters with the smallest and second smallest number of computing nodes to avoid a large number of idle links when crossing clusters, further optimizes the collective communication performance, and thus improves the distributed efficiency.

[0076] Based on the above embodiment, as an optional embodiment, Figure 6 As shown, if the gap between any two clusters in the target cluster and the unmerged cluster is not within a preset range, the method further includes: S64: Determine whether the difference between the number of computing nodes in the target cluster and the number of computing nodes in the specified cluster exceeds a threshold; wherein the specified cluster is the cluster with the smallest number of computing nodes in the unmerged clusters; if it exceeds the threshold, proceed to step S63; if it does not exceed the threshold, proceed to step S65.

[0077] Based on the above embodiment, after merging the smallest cluster and the second smallest cluster to obtain the target cluster, if the difference between the number of computing nodes of any two clusters in the target cluster and the unmerged cluster is still large, that is, after one merger, the difference in the number of computing nodes between the clusters is still large.

[0078] At this time, it is necessary to first determine in step S64 whether the reason for this is that the difference between the number of computing nodes in the target cluster and the number of computing nodes in the specified cluster exceeds the threshold. In other words, it is necessary to determine whether the number of computing nodes in the merged target cluster is too large, resulting in a large node gap between the current clusters.

[0079] If yes, that is, the difference between the number of computing nodes in the target cluster and the number of computing nodes in the specified cluster exceeds the threshold, then stop merging clusters and return to step S63, that is, perform data reduction with the currently merged target cluster and the unmerged cluster as new clusters participating in the reduction.

[0080] S65: Determine whether the number of unmerged clusters is greater than 1; if so, proceed to step S60 and execute subsequent steps; if not, proceed to the step of taking the target cluster and the unmerged cluster as clusters participating in data reduction and execute subsequent steps.

[0081] Of course, in another optional embodiment, if the difference between the number of computing nodes in the target cluster and the number of computing nodes in the specified cluster does not exceed the threshold, it means that the uneven number of computing nodes between the current clusters is not caused by the fact that the number of computing nodes in the merged target cluster is too large.

[0082] At this time, step S65 is further used to determine whether the number of unmerged clusters is greater than 1. In fact, after determining that the uneven nodes between the current clusters are not caused by the target cluster, clusters can continue to be merged to achieve the goal of even nodes between clusters.

[0083] Specifically, step S65 is used to determine whether the current target cluster and the unmerged clusters are still greater than 2, that is, to determine whether the number of current clusters can be further merged. If yes, return to step S60 to continue merging clusters. However, it should be noted that each time the step returns to step S60, the merged target cluster and the unmerged clusters are sorted and merged.

[0084] Of course, if the current number of clusters cannot be merged further, that is, the number of clusters that have not undergone data reduction is 1, that is, there are only 2 clusters left. At this time, the only way is to return to step S63 and use the target cluster and the unmerged cluster as clusters participating in data reduction; and enter step S10 in the above embodiment, and execute subsequent steps S11 and S12.

[0085] As an optional embodiment, the difference in the number of computing nodes between the clusters participating in data reduction is within a preset range, including: If the result of the specified calculation of the maximum and minimum numbers of computing nodes in the cluster participating in the data reduction is not greater than the preset value, the gap is within the preset range; wherein the specified calculation includes ratio calculation or difference calculation.

[0086] In a specific embodiment, when determining whether the difference in the number of computing nodes between clusters participating in data reduction is within a preset range, the number of computing nodes in each cluster may be counted, and a maximum number of computing nodes Umax and a minimum number of computing nodes Umin may be determined.

[0087] Further, a specified calculation is performed on the maximum value Umax and the minimum value Umin, and the specified calculation may include but is not limited to a ratio calculation or a difference calculation. When performing the ratio calculation, determine whether the calculation result of Umax / Umin is greater than 1. If it is not greater than 1, the difference in the number of computing nodes representing each cluster is within an acceptable range, and cross-cluster data reduction can be performed at this time. If it is greater than 1, the difference in the number of computing nodes representing each cluster is too large. In order to avoid wasting more communication link resources, the clusters need to be merged. For the specific merging method, refer to the description of the above embodiment.

[0088] In another optional embodiment, the difference between the maximum value Umax and the minimum value Umin may also be calculated. If the difference is smaller than a preset value, it indicates that the difference in the number of computing nodes of each cluster is within an acceptable range. Otherwise, it is considered that the difference is not within an acceptable range.

[0089] It should be noted that for different specified calculation methods, the corresponding preset values ​​are set differently. In addition, it should be noted that in addition to the calculation method of the ratio and difference of the difference in the number of computing nodes of each cluster within an acceptable range, other calculation methods can also be used, and this application does not limit this. For example, calculate the difference in the number of computing nodes between all clusters, and further determine whether the difference between any two differences is less than the preset value. If it is less than, it indicates that the difference in the number of computing nodes of each cluster is within an acceptable range. Otherwise, it is considered that it is not within an acceptable range.

[0090] In the above-mentioned embodiments, the collective communication control method for distributed training is described in detail. The present application also provides a corresponding embodiment of a collective communication control device for distributed training.

[0091] Figure 7 A schematic diagram of the structure of a distributed training collective communication control device provided in an embodiment of the present application, such as Figure 7 As shown, the device comprises: The cluster internal reduction module 70 is used to reduce the data distributed on all computing nodes in the first cluster to the designated computing nodes in the first cluster when the difference in the number of computing nodes between any two clusters in the clusters participating in the data reduction is within a preset range; wherein the number of the designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes in the clusters participating in the data reduction; A cross-cluster reduction module 71, used to control the designated computing node to perform data reduction with computing nodes in the second cluster; The data synchronization module 72 is used to control the designated computing node to synchronize data with other nodes in the first cluster except the designated computing node.

[0092] In addition, the collective communication control device for distributed training provided in the embodiment of the present application also includes: A node group division module is used to divide the computing nodes in the first cluster into m node groups; wherein each node group has k computing nodes, and k=floor(n / m), n is the number of computing nodes in the first cluster, and m is the number of computing nodes in the second cluster; A designated computing node selection module is used to select a computing node from each node group as a designated computing node; The cluster internal reduction submodule is used to reduce the data distributed on all computing nodes in the first cluster to the designated computing node.

[0093] A first sorting module, used to sort the computing nodes in the first cluster according to their device serial numbers from small to large; A computing node partitioning module is used to partition k computing nodes into node groups in sequence based on the sorting result, in a manner that each node group is partitioned one computing node at a time; The designated computing node determination module is used to select the computing node with the smallest device serial number in each node group as the designated computing node.

[0094] The first processing module is used to determine whether there is a target computing node that has not been grouped among the computing nodes in the first cluster; if not, reduce the data in each node group to the designated computing node in the corresponding group; if so, execute the step of reducing the data in each node group to the designated computing node in the corresponding group; and divide the data on the target computing node into m portions of data on an even basis, and send the m portions of data to different designated computing nodes in a one-to-one correspondence manner.

[0095] The second sorting module is used to sort the nodes in ascending order of the number of computing nodes when the gap is not within a preset range; A merging module, used for merging the first cluster and the second cluster into a target cluster based on the sorting result; The second processing module is used to determine whether the gap between any two clusters in the target cluster and the unmerged cluster is within a preset range; if so, the target cluster and the unmerged cluster are used as clusters participating in data reduction; and the step of reducing the data distributed on all computing nodes in the first cluster to the specified computing nodes in the first cluster is entered, and subsequent steps are executed.

[0096] The third processing module is used to determine whether the difference between the number of computing nodes in the target cluster and the number of computing nodes in the designated cluster exceeds a threshold; wherein the designated cluster is the cluster with the smallest number of computing nodes in the unmerged clusters; if it exceeds the threshold, the step of taking the target cluster and the unmerged cluster as clusters participating in data reduction is entered, and subsequent steps are executed; if it does not exceed the threshold, it is determined whether the number of unmerged clusters is greater than 1; if it is greater, the step of sorting in ascending order of the number of computing nodes is entered, and subsequent steps are executed; if it is not greater, the step of taking the target cluster and the unmerged cluster as clusters participating in data reduction is entered, and subsequent steps are executed.

[0097] Figure 8 A schematic diagram of the structure of a collective communication control device for distributed training provided by another embodiment of the present application is shown in FIG. Figure 8 As shown, the collective communication control device for distributed training includes: a memory 80 for storing a computer program; The processor 81 is used to implement the steps of the collective communication control method for distributed training mentioned in the above embodiment when executing the computer program.

[0098] The collective communication control device for distributed training provided in this embodiment may include but is not limited to a laptop computer or a desktop computer.

[0099] Among them, the processor 81 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 81 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processor, referred to as DSP), a field programmable gate array (Field-Programmable Gate Array, referred to as FPGA), and a programmable logic array (Programmable Logic Array, referred to as PLA). The processor 81 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (Central Processing Unit, referred to as CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 81 may be integrated with a graphics processing unit (Graphics Processing Unit, referred to as GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 81 may also include an artificial intelligence (Artificial Intelligence, referred to as AI) processor, which is used to process computing operations related to machine learning.

[0100] The memory 80 may include one or more computer-readable storage media, which may be non-transitory. The memory 80 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 80 is at least used to store the following computer program 801, wherein the computer program, after being loaded and executed by the processor 81, can implement the relevant steps of the collective communication control method for distributed training disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 80 may also include an operating system 802 and data 803, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 802 may include Windows, Unix, Linux, etc. Data 803 may include, but is not limited to, relevant data involved in the collective communication control method for distributed training, etc.

[0101] In some embodiments, the collective communication control device for distributed training may also include a display screen 82 , an input and output interface 83 , a communication interface 84 , a power supply 85 , and a communication bus 86 .

[0102] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the collective communication control device for distributed training, and may include more or fewer components than those shown in the figure.

[0103] The collective communication control device for distributed training provided in an embodiment of the present application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the collective communication control method for distributed training in the above-mentioned embodiment.

[0104] It should be noted that, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring these operations to be performed in the specific order shown or to be performed sequentially, or requiring all illustrated operations to be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

Claims

1. A collective communication control method for distributed training, characterized in that: The method comprises: When the difference in the number of computing nodes between any two clusters in the clusters participating in data reduction is within a preset range, the data distributed on all computing nodes in the first cluster are reduced to the designated computing nodes in the first cluster; wherein the number of the designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes in the clusters participating in data reduction; Controlling the designated computing node to perform data reduction with computing nodes in the second cluster; Control the designated computing node to synchronize data with other nodes in the first cluster except the designated computing node.

2. The collective communication control method for distributed training according to claim 1, characterized in that: The reducing the data distributed on all computing nodes in the first cluster to a designated computing node in the first cluster includes: Divide the computing nodes in the first cluster into m node groups; wherein each of the node groups has k computing nodes, and k=floor(n / m), n is the number of computing nodes in the first cluster, and m is the number of computing nodes in the second cluster; Selecting a computing node from each of the node groups as the designated computing node; The data distributed on all computing nodes in the first cluster are reduced to the designated computing node.

3. The collective communication control method for distributed training according to claim 2, characterized in that: The selecting a computing node from each of the node groups as the designated computing node comprises: Sort the computing nodes in the first cluster according to their device serial numbers from small to large; Based on the sorting result, the k computing nodes are divided into the node groups in sequence in a manner that each node group divides one computing node at a time; The computing node with the smallest device serial number in each of the node groups is used as the designated computing node.

4. The collective communication control method for distributed training according to claim 2, characterized in that: The reducing the data distributed on all computing nodes in the first cluster to the designated computing node includes: Determine whether there is a target computing node that has not been grouped in the computing nodes in the first cluster; If not, the data in each of the node groups is reduced to the designated computing nodes in the corresponding group; If so, execute the step of reducing the data in each of the node groups to the designated computing nodes in the corresponding group; and divide the data on the target computing node into m portions of data on average, and send the m portions of data to different designated computing nodes in a one-to-one correspondence manner.

5. The collective communication control method for distributed training according to claim 1, characterized in that: The method further comprises: When the gap is not within the preset range, the computing nodes are sorted in ascending order of number; Based on the ranking results, the first and second clusters are merged into the target cluster; Determine whether the gap between any two clusters in the target cluster and the unmerged clusters is within a preset range; If yes, the target cluster and the unmerged cluster are used as the clusters participating in data reduction; and the step of reducing the data distributed on all computing nodes in the first cluster to the designated computing nodes in the first cluster is entered, and subsequent steps are executed.

6. The collective communication control method for distributed training according to claim 5, characterized in that: If the gap between any two clusters in the target cluster and the unmerged cluster is not within the preset range, the method further includes: Determine whether the difference between the number of computing nodes in the target cluster and the number of computing nodes in a specified cluster exceeds a threshold; wherein the specified cluster is the cluster with the smallest number of computing nodes in the unmerged clusters; If the threshold is exceeded, the step of taking the target cluster and the unmerged cluster as the clusters participating in data reduction is entered, and subsequent steps are performed; If the threshold is not exceeded, determine whether the number of the unmerged clusters is greater than 1; if it is greater, enter the step of sorting the computing nodes in ascending order, and execute subsequent steps; if it is not greater, enter the step of taking the target cluster and the unmerged cluster as the clusters participating in data reduction, and execute subsequent steps.

7. The collective communication control method for distributed training according to claim 1, characterized in that: The difference in the number of computing nodes between clusters participating in data reduction is within the preset range, including: If the result of a specified calculation of the maximum and minimum numbers of computing nodes in the cluster participating in data reduction is not greater than a preset value, the gap is within the preset range; wherein the specified calculation includes a ratio calculation or a difference calculation.

8. A collective communication control device for distributed training, characterized in that: The device comprises: The cluster internal reduction module is used to reduce the data distributed on all computing nodes in the first cluster to the designated computing nodes in the first cluster when the difference in the number of computing nodes between any two clusters in the clusters participating in the data reduction is within a preset range; wherein the number of the designated computing nodes is the same as the number of computing nodes in the second cluster, and the second cluster is the cluster with the smallest number of computing nodes in the clusters participating in the data reduction; A cross-cluster reduction module, used to control the designated computing node and the computing nodes in the second cluster to perform data reduction; The data synchronization module is used to control the designated computing node to synchronize data with other nodes in the first cluster except the designated computing node.

9. A distributed training collective communication control device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the collective communication control method for distributed training described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the collective communication control method for distributed training described in any one of claims 1 to 7 are implemented.

Images