Data communication method, distributed training system, and related device

Abstract

Provided in the present application are a data communication method, a distributed training system, and a related device. The method comprises the following steps: on the basis of a bandwidth ratio between a first communication domain and a second communication domain and system structure information of a training system, a first node determining first communication traffic of a node performing communication in the first communication domain, and second communication traffic of a node performing communication in the second communication domain, such that the ratio of the first communication traffic to the second communication traffic is adapted to the bandwidth ratio; on the basis of the first communication traffic, determining a first communication policy used by the first communication domain; on the basis of the second communication traffic, determining a second communication policy used by the second communication domain; and on the basis of the first communication policy and the second communication policy, completing collective communication. Therefore, during collective communication, each node may make maximum use of the bandwidth of a first communication domain and the bandwidth of a second communication domain, a communication domain having a larger bandwidth may bear more data, and a communication domain having a smaller bandwidth bears less data, thereby avoiding the waste of bandwidth resources, and improving the communication efficiency of collective communication.

Description

Data communication method, distributed training system and related device

[0001] The present application claims priority to the Chinese patent application No. 202411319997.7, filed on September 20, 2024, entitled "Data communication method, distributed training system and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular to a data communication method, a distributed training system and related device. BACKGROUND

[0003] In a distributed training system, the training of an AI model is implemented by multiple nodes, so the communication strategy between nodes is crucial to improve the efficiency of model training. In order to improve the communication efficiency between nodes, multiple nodes are usually divided into multiple communication groups, the nodes in the same communication group communicate through a first communication domain, and the nodes in different communication groups communicate through a second communication domain, so that the data communication between nodes can be parallelized, thereby improving the communication efficiency.

[0004] Generally, if the number of nodes is sr, and the data on each node is divided into sr parts, then when parallel communication, each node will obtain sr / 2 parts of data from other nodes in the communication group through the first communication domain, and (sr / 2)-1 parts of data from other communication groups through the second communication domain, so the ratio of the communication volume between the first communication domain and the second communication domain is fixed as [(sr / 2) / (sr / 2)-1]. However, the ratio of bandwidths between the first communication domain and the second communication domain may not be [(sr / 2) / (sr / 2)-1], and it is possible that the bandwidth of the first communication domain is much larger than that of the second communication domain, or the bandwidth of the second communication domain is much larger than that of the first communication domain. In this case, the fixed communication volume ratio will cause a problem of bandwidth resource waste, thereby limiting the communication efficiency. SUMMARY

[0005] The present application provides a data communication method, a distributed training system and related device, which are used to solve the problem that the ratio of communication volumes between different communication domains is fixed, while the ratio of bandwidths is not fixed, making them difficult to adapt to each other, resulting in bandwidth resource waste and affecting the efficiency of collective communication.

[0006] In a first aspect, a data communication method is provided. The method is applied to a distributed training system, the distributed training system comprising a plurality of communication groups, each communication group comprising a plurality of computing nodes, the computing nodes in a same communication group performing data communication through a first communication domain, and the computing nodes in different communication groups performing data communication through a second communication domain. The method comprises the following steps: a first node in the plurality of computing nodes acquires a bandwidth ratio between the first communication domain and the second communication domain, and system structure information of the distributed training system; the first node obtains a first communication amount and a second communication amount based on the bandwidth ratio and the system structure information, wherein the first communication amount is a total amount of communication of the first node with other nodes in the communication group through the first communication domain, and the second communication amount is a total amount of communication of the first node with nodes in other communication groups through the second communication domain, wherein a ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio; the first node performs collective communication with the other nodes in the communication group according to the first communication amount through the first communication domain, and performs collective communication with the nodes in the other communication groups according to the second communication amount through the second communication domain, to obtain a collective communication result; and the first node performs distributed training of an artificial intelligence (AI) model based on the collective communication result.

[0007] By implementing the method described in the first aspect, when the distributed training system starts to train a model, if collective communication is needed between nodes, each node can determine a first communication amount of the node in the first communication domain and a second communication amount of the node in the second communication domain according to the bandwidth ratio between the first communication domain and the second communication domain and the system structure information of the training system, so that the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, and then the node performs communication in the two communication domains according to the determined first communication amount and second communication amount, so that the two communication domains can be maximally utilized. Even if the bandwidth changes, the node can adjust the communication amount according to the bandwidth ratio in time, so that the communication domain with larger bandwidth can undertake more data and the communication domain with smaller bandwidth can undertake less data, thereby avoiding waste of bandwidth resources and improving the communication efficiency of collective communication.

[0008] In a possible implementation, the ratio of the first communication amount and the second communication amount being adapted to the bandwidth ratio comprises: the ratio of the first communication amount and the second communication amount being the same as the bandwidth ratio; or the ratio of the first communication amount and the second communication amount being approximately the same as the bandwidth ratio; or a difference between the ratio of the first communication amount and the second communication amount and the bandwidth ratio being less than a threshold value. In short, the ratio of the first communication amount and the second communication amount being adapted to the bandwidth ratio can mean that the two are the same, that is, x=y; or the two are approximately the same, that is, x≈y; or the difference between the two is less than a threshold value, that is, x-y≤threshold value z.

[0009] It should be understood that the adaptation can be understood as very close between the two, and the equal case is the best case, in which the bandwidth can be maximally utilized, and the closer the traffic ratio to the bandwidth ratio, the higher the bandwidth utilization.

[0010] In a possible implementation, the first node determines at least one first communication node and first data to be sent by each first communication node based on the first traffic, determines at least one second communication node and second data to be sent by each second communication node based on the second traffic, the first communication node belongs to the same communication group as the first node, and the second communication node belongs to a different communication group from the first node, the first node obtains the first data from each first communication node through the first communication domain and obtains the second data from each second communication node through the second communication domain, and obtains the collection communication result.

[0011] In the above implementation, the first node determines a plurality of first communication nodes to be communicated and first data to be obtained from each first communication node based on the first traffic, and determines a plurality of second communication nodes to be communicated and second data to be obtained from each second communication node based on the second traffic, so that when the collection communication is performed, the first data can be obtained from each first communication node through the first communication domain and the second data can be obtained from each second communication node, so that the collection communication can ensure that the total communication amount of the first communication domain is the first traffic and the total communication amount of the second communication domain is the second traffic, and ensure that the traffic ratio is adapted to the bandwidth ratio, so that the bandwidth utilization of the communication domain is maximized, thereby improving the efficiency of the collection communication.

[0012] In a possible implementation, the data on each node of the plurality of computing nodes is divided into a plurality of sub-data according to the total number of nodes, each sub-data corresponds to a data number, and the collection communication result includes sub-data with a first number on each node, and the first data and the second data include sub-data with the first number.

[0013] In a specific implementation, the unit of the traffic is the number or the fraction of the sub-data, where the data on each node is divided into rs fractions according to the total number of nodes rs, for example, the number of nodes rs = 8, and the data on each node is divided into 8 sub-data, node 0 includes sub-data A0-A7, node 1 includes sub-data B0-B7, and so on. If the first traffic is 3, it means that the first node obtains 3 fractions of sub-data from other nodes in the first communication group, and the second traffic is 4, which means that the first node obtains 4 fractions of sub-data from nodes in the other communication group. The above examples are used for illustration, and the application is not limited in this way.

[0014] In a specific implementation, the first node needs to obtain data on each of the other nodes, so the total amount of the first traffic and the second traffic is fixed. If the number of nodes in the communication group is M, the number of communication groups is N, the total number of nodes is rs=MN, and the data on each node is divided into rs parts, then the total amount of the first traffic and the second traffic is MN-1. If the first traffic is pn and the second traffic is hn, then pn+hn=rs-1, and the ratio between the first bandwidth and the second bandwidth is R, the ratio between the first traffic and the second traffic is pn / hn=R. Then, based on the binary linear equation set, the values of the first traffic pn and the second traffic hn can be obtained.

[0015] The above implementation divides the data on the nodes according to the total number of nodes, allocates a data number to each sub-data, and then performs collective communication in units of the number or the fraction of the sub-data, so that the data can be transmitted in fragments during collective communication, parallel transmission is achieved, transmission efficiency is improved, bandwidth is better utilized, and network congestion caused by excessively large data is avoided. At the same time, the amount of traffic is calculated based on the number or the fraction of the sub-data, so that the size of the traffic can be better calculated.

[0016] In a possible implementation, the first node determines at least one first communication node based on system structure information, wherein the first communication node includes all nodes in the communication group in which the first node is located except the first node, and the first node determines first data to be sent by each first communication node based on the first traffic, wherein the first data includes one or more sub-data, and the difference between the data number of each sub-data and the first number is a multiple of M, where M is the number of nodes in each communication group.

[0017] In a specific implementation, the first data can include a first sub-data, and the number of the first sub-data is the number of the sub-data required by the collective communication result of the first node. For example, the collective communication result of the first node is A0+B0+…, that is, the first part of the sub-data on each node, so the number of the first sub-data is 0. If the collective communication result of the first node is A5+B5+…, that is, the sixth part of the sub-data on each node, then the number of the first sub-data is 5. In general, if the node number and the sub-data number use the same encoding method, such as using numbers to represent the node number and the sub-data number, then the node number of the first node and the number of the first sub-data are the same. For example, node 1 needs to obtain the sub-data numbered 1 on all nodes, so the number of the first sub-data is 1, and node 5 needs to obtain the sub-data numbered 5 on all nodes, so the number of the first sub-data is 5.

[0018] Further, the first data can further include one or more second sub-data, and a difference between the number of the first sub-data and the number of the second sub-data is the number M of nodes in the communication group or a multiple of M. For example, the number of the first sub-data is 0, and the number M of nodes in the communication group is 4, and then the number of the second sub-data can be 4, 8, 12, and the like. The number of the second sub-data can be determined according to the first communication amount. If the first communication amount is pn, and the number of the first communication nodes is M-1, then the number of the sub-data included in each first data is sn= pn / (M-1), and the sum of the number of the first sub-data and the number of the second sub-data is sn, and therefore the number of the second sub-data is sn-1.

[0019] It should be noted that the total number of nodes rs= MN, and the maximum number of sub-data on each node is MN. If the number of the second sub-data calculated according to the multiple of M is greater than MN, the excess part can be calculated from 0 to obtain the actual number of the second sub-data. For example, the number of the second sub-data is calculated to be 9, but the total number of nodes is 8, and the excess part is 1, which is calculated from 0 to obtain the actual number of the second sub-data as 1. For another example, the number of the second sub-data is calculated to be 15, but the total number of nodes is 12, and the excess part is 3, which is calculated from 0 to obtain the actual number of the second sub-data as 3. The above examples are used for illustration, and the present application is not limited in this way.

[0020] In the above implementation, the first communication node in the first communication domain is determined, that is, all nodes in the communication group except the first node, but the first data to be sent by each first communication node is uncertain. Based on the calculated first communication amount and the number of the first communication nodes, and in combination with the result required by the set communication, the data number of the first data to be sent by each first communication node can be calculated, so that when the first node communicates based on the first communication domain, the total communication amount can reach the first communication amount, and further, the bandwidth ratio of the two communication domains is adapted to the ratio of the communication amounts.

[0021] In a possible implementation, the first node determines the second data, and the second data includes sub-data with a first number. The first node determines the number h1 of the full communication nodes and the number h2 of the single communication nodes based on the second communication amount and the number N of the communication groups, wherein the second communication nodes include the full communication nodes and the single communication nodes. All nodes in the communication group where the full communication nodes are located communicate with the first node, and only the single communication nodes in the communication group where the single communication nodes are located communicate with the first node. The first node determines at least one second communication node based on the number h1 of the full communication nodes and the number h2 of the single communication nodes in combination with the system structure information.

[0022] That is, in the second communication domain, the first node needs to obtain the first sub-data from each second communication node, so that the first node can obtain the final set communication result in combination with the first sub-data obtained in the first communication domain. As described above, the first sub-data number is the number of sub-data required for the set communication result of the first node, such as the set communication result of the first node is A0+B0+…, and the first sub-data number is 0. Therefore, in the second communication strategy, the data number of the second data is determined, that is, the number of the first sub-data, and the node number of the second communication node needs to be calculated.

[0023] Further, the full communication node refers to all nodes in a whole communication group that need to communicate with the first node, and the single communication node refers to only one node in a whole communication group that needs to communicate with the first node. The total amount of communication of the full communication node and the single communication node is determined according to the second communication amount, the total number of nodes of the full communication node and the single communication node is determined according to the number of communication groups N, and the number of full communication nodes h1 and the number of single communication nodes h2 are determined based on the total amount of communication and the total number of nodes. Then, each second communication node is determined in combination with system structure information.

[0024] That is, if the number of nodes in a communication group is M, the second communication amount is hn (the first communication amount pn and the second communication amount hn can be determined based on the description above), the number of full communication nodes is h1, and the number of single communication nodes is h2, then the second communication amount hn=h1+h2. In combination with the number of communication groups N, the first node needs to communicate with each communication group, and h1 / M+h2=N-1 can be determined. Based on the binary linear equation system, the number of full communication nodes h1 and the number of single communication nodes h2 can be calculated. Then, each second communication node is determined in combination with system structure information.

[0025] The above implementation mode, for the second communication domain, the second data to be communicated is fixed, that is, each second communication node sends the first number of sub-data to the first node, because the set communication result of the first node is the first number of sub-data. Then the main calculation is which each second communication node is, the present application classifies the second communication nodes, each communication group either all communicates with the first node, that is, the full communication node, or only one node communicates with the first node, that is, the single communication node, so that the number of single communication nodes and full communication nodes in the second communication node can be calculated by a simple binary linear equation. Then, according to the system structure, the communication group that needs single communication and the communication group that needs full communication can be determined, and the second communication node can be determined. The way to determine the second communication node is simple and efficient, which improves the calculation efficiency.

[0026] In a possible implementation, in a possible implementation, the first node determines all nodes in h1 / M communication groups after the first communication group in which the first node is located as full communication nodes, and the first node determines the i th node in h2 communication groups after the communication group in which the last full communication node is located as a single communication node, wherein the first node is the i th node in the first communication group.

[0027] Briefly, the first communication group in which the first node is located is determined, and then h1 nodes after the first communication group are determined as full communication nodes, and the node number of each full communication node can be determined in the order of numbering. Then h2 single communication nodes are determined from the remaining h2 communication groups, one single communication node from each communication group, and the single communication node of each communication group is in the same position in the numbering order of the multiple nodes in the communication group, that is, the numbering order of the first node in the first communication group.

[0028] The numbering order position refers to the ordering position of the nodes in the communication group after ordering by number size. Assuming that the first node is the i th node in the first communication group, the full communication node is also the i th node in the communication group. For example, the total number of nodes is 12, the number of communication groups is 3, and each communication group includes 4 nodes. The first communication group includes nodes 0, 1, 2, and 3, the second communication group includes nodes 4, 5, 6, and 7, and the third communication group includes nodes 8, 9, 10, and 11. The numbering order positions of nodes 0, 4, and 8 are the same, and they are all the first nodes in the numbering order in the communication group. The numbering order positions of nodes 1, 5, and 9 are the same, and they are all the second nodes in the numbering order in the communication group. The numbering order positions of nodes 2, 6, and 10 are the same, and they are all the third nodes in the numbering order in the communication group. The numbering order positions of nodes 3, 7, and 11 are the same, and they are all the fourth nodes in the numbering order in the communication group. Therefore, if the first node is node 0, only nodes with numbering order positions 1, 5, and 9 are the same as the first node, and only these three nodes can be used as single communication nodes. The above example is used for illustration, and the present application is not limited in this regard.

[0029] The above implementation, based on the rules of set communication, allocates full communication nodes and single communication nodes according to node order, so that the result of set communication is not affected after the communication capacity is allocated according to the technical solution of the present application, and the ratio of bandwidth to communication capacity is also adapted, so that the bandwidth utilization rate is maximized, the communication efficiency is improved, and the result of set communication is also accurate.

[0030] In a possible implementation, the first node matches the bandwidth ratio and the system structure information with the mapping relationship, and determines the first communication node, the first data, the second communication node and the second data based on a communication strategy corresponding to the first bandwidth ratio and the system structure information in the case that the mapping relationship includes the system structure information and the gap between the bandwidth ratio and the first bandwidth ratio in the mapping relationship is less than a threshold. In the case that the mapping relationship does not include the system structure information, or the mapping relationship includes the system structure information but does not include the first bandwidth ratio, the first node obtains the first communication amount and the second communication amount based on the bandwidth ratio and the system structure information.

[0031] In short, the first node can also calculate the communication strategy corresponding to each system structure under different bandwidth ratios according to the method described above, the communication strategy including the first communication node, the first data, the second communication node and the second data, and then determine the communication strategy mapped by different system structures and different bandwidth ratios. In this way, if the current bandwidth ratio of the first node is constantly changing, the corresponding communication strategy can be constantly updated according to the mapping relationship, the repeated calculation is reduced, and the efficiency of obtaining the communication strategy is improved.

[0032] The above implementation records the mapping relationship between the calculated communication strategy and the bandwidth ratio and the system structure information, so that the first node can determine whether there is a corresponding communication strategy according to the mapping relationship after obtaining the current bandwidth ratio and the system structure information. If there is, the communication strategy determined by the mapping relationship can be directly used. If not, the communication amount and the communication node and the data to be communicated corresponding to the communication amount are calculated by the method described above, the repeated calculation is reduced, and the communication efficiency is further improved. In this way, during model training, no matter how the network fluctuates, the best communication strategy can be determined according to the current bandwidth when the nodes perform collective communication, so that the bandwidth of the two communication domains can be maximized, and the problem of resource waste does not occur, the efficiency of collective communication is improved, and the overall training efficiency of the model is improved.

[0033] In a possible implementation, the collective communication method includes one or more of a reducescatter, an allreduce, an allgather and an all-to-all algorithm.

[0034] It should be understood that the above mainly takes the reducescatter algorithm as an example to introduce the data communication method of the application, that is, the final set communication result includes the first numbered sub-data, and for other set communication algorithms, such as allgather and allreduce algorithms, the communication amount and communication strategy of each communication domain can be determined according to the set communication result required by each node, combined with the system structure information and the bandwidth ratio, so that the ratio of the communication amount is adapted to the bandwidth ratio, and here, each set communication algorithm is not illustrated one by one.

[0035] It should be noted that the data communication method provided by the application can be applied to set communication between nodes. Generally, set communication occurs in the model training phase, and using the data communication method provided by the application can improve the training efficiency of the model. In some special cases, set communication can also occur in the model inference phase, and using the data communication method provided by the application can also improve the inference efficiency of the model. Here, examples are not illustrated one by one.

[0036] In a second aspect, a computing node is provided. The computing node is a computing node in a distributed training system. The distributed training system includes a plurality of communication groups. Each communication group includes a plurality of computing nodes. The computing nodes in the same communication group perform data communication through a first communication domain. The computing nodes in different communication groups perform data communication through a second communication domain. The computing node includes: an obtaining unit configured to obtain a bandwidth ratio between the first communication domain and the second communication domain, and system structure information of the distributed training system; a calculating unit configured to obtain a first communication amount and a second communication amount based on the bandwidth ratio and the system structure information, wherein the first communication amount is a total communication amount of the first node communicating with other nodes in the communication group through the first communication domain, the second communication amount is a total communication amount of the first node communicating with nodes in other communication groups through the second communication domain, and a ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio; a communicating unit configured to perform set communication with other nodes in the communication group according to the first communication amount based on the first communication domain, and perform set communication with nodes in other communication groups according to the second communication amount based on the second communication domain, to obtain a set communication result; and a training unit configured to perform distributed training of an artificial intelligence (AI) model based on the set communication result.

[0037] The computing node implementing the second aspect, when the distributed training system starts to train the model in a distributed manner, if the nodes need to perform the collective communication, each node can determine, according to the bandwidth ratio between the current first communication domain and the second communication domain and the system structure information of the training system, a first communication amount of the node in the first communication domain and a second communication amount of the node in the second communication domain, so that the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, and then performs communication in the two communication domains according to the determined first communication amount and second communication amount, so that both communication domains can be maximally utilized, that is, even if the bandwidth changes, the nodes can adjust the communication amount in time according to the bandwidth ratio, so that the communication domain with large bandwidth can bear more data, and the communication domain with small bandwidth can bear less data, thereby avoiding waste of bandwidth resources and improving the communication efficiency of the collective communication.

[0038] In a possible implementation, the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, including that the ratio of the first communication amount and the second communication amount is the same as the bandwidth ratio, or the ratio of the first communication amount and the second communication amount is approximately the same as the bandwidth ratio, or the difference between the ratio of the first communication amount and the second communication amount and the bandwidth ratio is less than a threshold value.

[0039] In a possible implementation, the computing unit is configured to determine, based on the first communication amount, at least one first communication node and first data to be sent by each first communication node, and determine, based on the second communication amount, at least one second communication node and second data to be sent by each second communication node, wherein the first communication node belongs to the same communication group as the first node, and the second communication node belongs to a different communication group from the first node, and the communication unit is configured to obtain the first data from each first communication node through the first communication domain, and obtain the second data from each second communication node through the second communication domain, to obtain the collective communication result.

[0040] In a possible implementation, the data on each node of the plurality of computing nodes is divided into a plurality of sub-data according to the total number of nodes, each sub-data corresponds to a data number, and the collective communication result includes sub-data with a first number on each node, and the first data and the second data include sub-data with the first number.

[0041] In a possible implementation, the computing unit is configured to determine, based on the system structure information, at least one first communication node, wherein the first communication node includes all nodes in the communication group in which the first node is located except the first node, and determine, based on the first communication amount, first data to be sent by each first communication node, wherein the first data includes one or more sub-data, and the difference between the data number of each sub-data and the first number is a multiple of M, and M is the number of nodes in each communication group.

[0042] In a possible implementation, the computing unit is configured to determine second data, the second data comprising sub-data numbered as the first number, determine the number h1 of full communication nodes and the number h2 of single communication nodes based on the second communication volume and the number N of communication groups, wherein the second communication nodes comprise the full communication nodes and the single communication nodes, all nodes in the communication group where the full communication node is located communicate with the first node, only the single communication node in the communication group where the single communication node is located communicates with the first node, and determine at least one second communication node based on the number h1 of full communication nodes and the number h2 of single communication nodes and in combination with the system structure information.

[0043] In a possible implementation, the computing unit is configured to determine that all nodes in the h1 / M communication groups after the first communication group where the first node is located are full communication nodes, and determine that the i-th node in the h2 communication groups after the communication group where the last full communication node is located is a single communication node, wherein the first node is the i-th node in the first communication group.

[0044] In a possible implementation, the computing unit is configured to match the bandwidth ratio and the system structure information with a mapping relationship, determine the first communication node, the first data, the second communication node, and the second data based on a communication strategy corresponding to the first bandwidth ratio and the system structure information in a case where the mapping relationship comprises the system structure information and a gap between the bandwidth ratio and the first bandwidth ratio in the mapping relationship is less than a threshold, and obtain the first communication volume and the second communication volume based on the bandwidth ratio and the system structure information in a case where the mapping relationship does not comprise the system structure information or the mapping relationship comprises the system structure information but does not comprise the first bandwidth ratio.

[0045] In a possible implementation, the algorithm used by the collective communication comprises one or more of a reducescatter algorithm, an allreduce algorithm, an allgather algorithm, and an all-to-all algorithm.

[0046] In a third aspect, a computing device is provided, which comprises a processor and a memory, the memory is configured to store instructions, and the processor is configured to execute the instructions to enable the computing device to implement the method described in the first aspect.

[0047] In a fourth aspect, a distributed training system is provided, which includes a plurality of communication groups, each of which includes a plurality of computing devices, the computing devices in the same communication group communicate with each other through a first communication domain, and the computing devices in different communication groups communicate with each other through a second communication domain, each of the computing devices includes a processor and a memory, and the processor of each of the computing devices is configured to execute instructions stored in the memory, so that each of the computing devices implements the method described in the first aspect.

[0048] In a fifth aspect, a computer readable storage medium is provided, which stores instructions, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0049] In a sixth aspect, a computer program product is provided, which includes instructions, and the instructions can be executed on a computing device or stored in any available medium or software or program product, and when the computer program product is executed on a computing device or a cluster of computing devices, the computing device or the cluster of computing devices executes the method described in the first aspect.

[0050] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0051] FIG. 1 is a schematic diagram of steps of a distributed system implementing ReduceScatter collective communication through layered communication;

[0052] FIG. 2 is an architecture diagram of a training system provided by the present application;

[0053] FIG. 3 is an example diagram of a training system 200 provided by the present application deployed in a cloud environment;

[0054] FIG. 4 is a flowchart of collective communication of each node in a training system provided by the present application;

[0055] FIG. 5 is a schematic diagram of steps of a data communication method provided by the present application;

[0056] FIG. 6 is an example diagram of a communication strategy of a data communication method provided by the present application in a first application scenario;

[0057] FIG. 7 is an example diagram of a communication strategy of a data communication method provided by the present application in a second application scenario;

[0058] FIG. 8 is an example diagram of a communication strategy of a data communication method provided by the present application in a third application scenario;

[0059] FIG. 9 is an example diagram of a communication strategy of a data communication method provided by the present application in a fourth application scenario;

[0060] FIG. 10 is an example diagram of a communication strategy of a data communication method provided by the present application in a fifth application scenario;

[0061] FIG. 11 is an example diagram of a communication strategy of a data communication method provided by the present application in a sixth application scenario;

[0062] FIG. 12 is an example diagram of a communication strategy of a data communication method provided by the present application in a seventh application scenario;

[0063] FIG. 13 is a structural schematic diagram of a computing node provided by the present application;

[0064] FIG. 14 is a structural schematic diagram of a computing device provided by the present application;

[0065] FIG. 15 is an example diagram of a computing device cluster provided by the present application. DETAILED DESCRIPTION

[0066] First, the application scenarios involved in the present application are explained and described.

[0067] With the continuous development of AI technology, the training of AI models begins to rely on greater computing power and storage capacity, and a single node (also referred to as a computing node) gradually cannot meet the training needs of AI models, so super-large-scale AI models usually use distributed training to complete. Distributed training cuts the model parameters and distributes them to multiple nodes for training, which not only reduces the computing pressure of a single node, but also allows multiple nodes to train in parallel, improving training efficiency. In the process of distributed training, each node is responsible for updating part of the parameters of the model, but the goal of training is to train a complete model, so communication is needed between nodes to synchronize the data trained by each node to other nodes through collective communication.

[0068] Among them, collective communication is a common communication method of distributed systems, which is a global communication operation in which all nodes in the system participate. In general, collective communication can include Allgather communication, Allreduce communication, Reducescatter communication, all-to-all communication, etc., and can also include more types of collective communication, which are not illustrated one by one here. Allgather communication means that the data of each node is summarized on each node, so that each node includes the data of all nodes. Allreduce communication means that each node obtains the data on other nodes. ReduceScatter communication means that all nodes perform the same reduction operation and then distribute the operation results to each node.

[0069] For example, if the node participating in the distributed training is 4, node 0 includes data A, node 1 includes data B, node 2 includes data C, and node 3 includes data D, after Allreduce communication, the data on each node is the same, that is, data X, which is the sum of data A, data B, data C and data D. After Allgather communication, the data on each node is the same, that is, data A, data B, data C and data D. After ReduceScatter communication, data A is evenly divided into A0-A3, data B is evenly divided into B0-B3, data C is evenly divided into C0-C3, and data D is evenly divided into D0-D3. After communication, node 0 includes data A0+B0+C0+D0, node 1 includes data A1+B1+C1+D1, node 2 includes A2+B2+C2+D2, and node 3 includes A3+B3+C3+D3. It should be understood that the above examples are for illustration.

[0070] The above various collective communication methods all require simultaneous communication between multiple nodes, and the bandwidth between nodes is limited. In order to improve the efficiency of communication, a layered communication method is usually used to implement it. Specifically, a plurality of nodes are divided into a plurality of communication groups, nodes in the same communication group communicate using a first communication domain, and nodes in different communication groups communicate using a second communication domain, so that the data communication between nodes can be parallel, thereby improving the efficiency of collective communication.

[0071] For example, FIG. 1 is a step flow diagram of a distributed system implementing ReduceScatter collective communication through a layered communication method. As shown in FIG. 1, the distributed system includes 8 nodes, node 0 to node 7, node 0 stores data A, node 1 stores data B, and so on. The data on each node is divided into 8 parts, for example, node 0 includes data A0-A7, node 1 includes data B0-B7, and so on.

[0072] Further, the 8 nodes are divided into 4 communication groups, each communication group including 2 nodes. In FIG. 1, the two nodes in the dashed box belong to the same communication group, that is, node 0 and node 1 belong to the first communication group, node 2 and node 3 belong to the second communication group, node 4 and node 5 belong to the third communication group, and node 6 and node 7 belong to the fourth communication group. The two nodes in the group communicate with each other through a first communication method, which is called a communication domain. The two nodes in different groups communicate with each other through a second communication method, which is called a communication domain. The following will explain in detail how to implement ReduceScatter collective communication in this application scenario in combination with FIG. 1.

[0073] Firstly, each communication group communicates within the group through the first communication domain to realize intra-group data synchronization, wherein each node in the group sends 4 pieces of data to another node, so that both sides in the group obtain the data of 2 nodes in the communication group, and FIG. 1 distinguishes the two communication directions within the group by solid lines and dashed lines, for example, node 0 sends A1, A3, A5 and A7 to node 1, and node 0 receives B0, B2, B4 and B8 sent by node 1, and by analogy, the nodes of other communication groups will perform the same operation, which will not be repeated here.

[0074] Secondly, while continuously synchronizing data within the group, the communication groups communicate between groups through the second communication domain to realize inter-group data synchronization, wherein each node continuously sends the 2 nodes of data obtained by intra-group synchronization to the other 3 communication groups, so that the other communication groups can obtain the data of all 8 nodes. FIG. 1 uses solid lines to represent the data acquisition direction of node 1 between communication groups, that is, to acquire data from node 3, node 5 and node 7, and uses dashed lines to represent the data acquisition direction of node 0, that is, to acquire data from node 2, node 4 and node 6, and the data acquisition direction of other communication groups can be referred to node 0 and node 1, which will not be repeated here.

[0075] Taking node 1 as an example, node 1 receives data A1 sent by node 0 through the first communication domain, node 3 receives data C1 sent by node 2 through the first communication domain, node 5 receives data F1 sent by node 4 through the first communication domain, and node 7 receives data H1 sent by node 6 through the first communication domain. Then node 1 receives C1+D1 sent by node 3 through the second communication domain, receives E1+F1 sent by node 5 through the second communication domain, and receives G1+H1 sent by node 7 through the second communication domain. In this way, after node 1 performs ReduceScatter collective communication, it will obtain data A1+B1+C2+…+H1, and by analogy, other nodes can also obtain the results shown in FIG. 1 in the above manner.

[0076] It should be understood that because the communication is bidirectional, each node in the group obtains 4 pieces of data from another node when communicating within the group, that is, the communication volume of communication domain 1 is 4 pieces of data, and each node in the group obtains 3 pieces of data from other communication groups when communicating between groups, that is, the communication volume of communication domain 2 is 3 pieces of data, so the ratio of the communication volume of communication domain 1 to the communication volume of communication domain 2 is 4 to 3. However, in actual business scenarios, the bandwidth ratio of communication domain 1 to communication domain 2 is not 4 to 3, and the bandwidth of communication domain 1 may be much smaller than that of communication domain 2, or the bandwidth of communication domain 1 may be much larger than that of communication domain 2, thereby causing waste of bandwidth resources.

[0077] Meanwhile, even if the bandwidth ratio of the first communication domain and the second communication domain is exactly 4 to 3, the bandwidth of the first communication domain and the second communication domain may fluctuate in different scenarios under the same training system. The bandwidth ratio may be 4 to 3 before, and then change due to network load and other reasons, so that the bandwidth resource is wasted when the same communication is performed according to the communication amount of 4 to 3.

[0078] In summary, in the current distributed system, if the layered communication is used to realize the collective communication when training the model, the communication amount ratio of the two communication domains is fixed. If the number of nodes is sr, and the data on each node is divided into sr parts, then the communication amount ratio of the communication domain 1 and the communication domain 2 is fixed as [(sr / 2) / (sr / 2)-1] when parallel communication is performed. However, the bandwidth ratio of the communication domain 1 and the communication domain 2 may not be [(sr / 2) / (sr / 2)-1], and the bandwidth of the communication domain 1 may be much larger than the bandwidth of the communication domain 2, or the bandwidth of the communication domain 1 may be much smaller than the bandwidth of the communication domain 2. In this case, the fixed communication amount ratio will cause the problem of bandwidth resource waste, affect the communication efficiency of the distributed system, and further affect the training efficiency of the model.

[0079] In order to solve the problem that the communication amount ratio of different communication domains is fixed when the current distributed system trains the model, the bandwidth resource is wasted, the collective communication efficiency is affected, and the model training efficiency is affected, the present application provides a data communication method. The method is applied to a distributed training system. When the training system starts to train the model in a distributed manner, if the nodes need to perform collective communication, each node can determine the first communication amount of the node in the first communication domain and the second communication amount of the node in the second communication domain according to the bandwidth ratio between the first communication domain and the second communication domain and the system structure information of the training system, so that the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio. Then, the first communication strategy used by the first communication domain is determined based on the first communication amount, the second communication strategy used by the second communication domain is determined based on the second communication amount, and then the collective communication is completed based on the first communication strategy and the second communication strategy. In the collective communication process, each node can maximize the utilization of the bandwidth of the first communication domain and the second communication domain. Even if the bandwidth fluctuates, the node can adjust the communication strategy in time, so that the communication domain with larger bandwidth can bear more data, and the communication domain with smaller bandwidth can bear less data, thereby avoiding the waste of bandwidth resource and improving the communication efficiency of the collective communication.

[0080] FIG. 2 is an architecture diagram of a training system provided by the present application, as shown in FIG. 2, the architecture includes a client 100 and a training system 200, the training system 200 can be a distributed training system for training an AI model by a distributed training method, the client 100 and the training system 200 are in communication connection, which can be wired network connection or wireless network connection, the wired network connection includes Ethernet, occupied line, cable, etc., the wireless network connection includes wireless fidelity (Wi-Fi), cellular network, etc., and can also include internet, local area network (LAN) and the like which support wired network connection and wireless network connection at the same time, and of course can also include other network types, which are not limited by the present application. The number of clients 100 can be one or more, and FIG. 2 takes one client as an example for illustration, which is not limited by the present application.

[0081] The client 100 is deployed on a terminal device or a computing device, and is used to realize human-computer interaction. The terminal device includes a personal computer, a smart phone, a wearable device, a palm-held processing device, a tablet computer, a mobile notebook, an augmented reality (AR) device, a virtual reality (VR) device, a smart conference device, and the like, which are not limited by the present application. The computing device can be a bare metal server (BMS), a virtual machine or a container. The BMS refers to a general-purpose physical server, for example, an ARM server or an X86 server; the virtual machine refers to a complete computer system that is simulated by software and runs in a completely isolated environment, and has complete hardware system functions. The work that can be completed in a physical computer can also be realized in the virtual machine. When creating a virtual machine in a computing device, part of the hard disk and memory capacity of the physical machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has independent basic input / output system (BIOS), hard disk and operating system, and can be operated like a physical machine; the container is a portable software unit, which can combine an application and all its dependencies into a software package, and the software package is not limited by the underlying host operating system, so that a complex environment does not need to be built again, and the process of application development to deployment is simplified.

[0082] The training system 200 can be a computing device, or a computing device cluster composed of multiple computing devices, and the description of the computing device can refer to the above description, which is not repeated here.

[0083] Optionally, the client 100 and the training system 200 can be deployed on the same computing device, or the client 100 and the training system 200 can be deployed on different computing devices in the same computing device cluster, or the client 100 and the training system 200 can be deployed on different computing devices in different computing device clusters, and the present application does not make specific limitations.

[0084] Optionally, the client 100 can be a terminal device or a software or application program running on a computing device controlled by a user, such as a personal computer (PC) client, a world wide web (web) client based on browser access, an application (APP) client running on a mobile terminal, or a console of a cloud platform, and the present application does not make specific limitations. The user holding the client 100 can be a developer of a training model, such as a programmer or an AI engineer, and the present application does not make specific limitations.

[0085] Optionally, the client 100 can be a client of a model service platform or a machine learning framework for processing machine learning related tasks, which provides a series of tools and services for the whole process of definition, training, deployment, inference, management, etc. of a machine learning model. The model service platform can be ModelArts, OpenAI, etc., and the machine learning framework can be TensorFlow, PyTorch, MindSpore, etc., and the present application does not make specific limitations.

[0086] Optionally, the client 100 can also be a client of a cloud platform for users to purchase and rent various cloud services. The data communication method provided by the present application can be one of the cloud services, and the user can purchase the cloud service separately to optimize the efficiency of the collection communication in the model training process. Alternatively, the cloud platform provides a comprehensive service to the user, and the data communication method provided by the present application can be a sub-service of the comprehensive cloud service. For example, the comprehensive cloud service is an AI cloud service, and the data communication method provided by the present application can be a sub-service of the AI cloud service for optimizing the efficiency of the collection communication in the model training process.

[0087] The possible deployment modes of the training system 200 and the client 100 are described in detail above, and in actual deployment, flexible deployment can be performed in combination with specific application scenarios and business requirements. The actual deployment modes of the training system 200 and the client 100 are described below by way of example.

[0088] For example, the training system 200 and the client 100 can be deployed in an internal network of an enterprise, the training system 200 is deployed in a server or a server cluster purchased by the enterprise, and the client 100 is a remote desktop connection tool or a management console software of a server. Employees of the enterprise can remotely connect to the server or the server cluster where the training system 200 is located through the client 100, and use the training system 200 to optimize the collection communication efficiency in the model training process.

[0089] For another example, the training system 200 can be deployed in a cloud environment. For example, FIG. 3 is an example diagram of a training system 200 deployed in a cloud environment according to the present application. As shown in FIG. 3, a user can initiate a purchase request for an AI cloud service through a client 100, and the AI cloud service includes a sub-service corresponding to the data communication method provided by the present application. Specifically, after the client 100 sends the purchase request to the cloud platform, the cloud platform can provide the client 100 with the use right of the training system 200 provided by the present application, so that the user can use the training system 200 through the client 100 to optimize the collection communication efficiency in the model training process.

[0090] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, security resources, etc., to meet the computing needs of the training system 200 under different scales and loads. These computing resources can be dynamically scaled according to the use needs of the training system 200, to ensure the stable operation of the training system 200 and provide reliable AI cloud services for users.

[0091] It should be understood that the above application scenarios are used for illustration, and the training system 200 and the client 100 can be flexibly trained according to actual business needs, which will not be illustrated one by one here.

[0092] Further, the training system 200 includes a plurality of communication groups 210, and each communication group includes a plurality of nodes (or computing nodes). The number of nodes in each communication group 210 and the number of communication groups 210 in the training system 200 are not limited by the present application.

[0093] The node can be a computing unit or a computing device with data processing capability, which can include a processor implemented by a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), or any combination thereof, without limitation. The computing unit can also include an acceleration card that has good performance in executing machine learning tasks and can improve the speed and efficiency of neural network training, such as a graphics processing unit (GPU), a neural network processing module (NPU), a data processing unit (DPU), a tensor processing unit (TPU), or the like. The computing device can be a BMS, a virtual machine, or a container, which can be referred to in the foregoing, and will not be repeated here.

[0094] The node can be held by a user or provided by a cloud service provider to the user. When the node is provided by the cloud service provider to the user, the node can also be referred to as an instance. It should be understood that the cloud service provider can provide a GPU instance, a TPU instance, a virtual machine instance, or a container instance dedicated to machine learning to the user. The user can purchase the use right of the instance from the cloud service provider according to the use demand, without limitation.

[0095] In the embodiments of the present application, a plurality of communication domains are established between a plurality of nodes by different communication modes. A plurality of nodes that establish a communication connection by using the same communication mode can form a communication domain. For example, FIG. 2 illustrates two communication domains. In the example shown in FIG. 2, a plurality of nodes can be divided into a plurality of communication groups. A plurality of nodes in each communication group establish a first communication domain through a first communication connection. Nodes in different communication groups establish a second communication domain through a second communication connection. The first communication connection and the second communication connection are different communication modes.

[0096] Optionally, each node in the first communication domain is directly connected with all other nodes. Each node in the second communication domain can be directly connected with all other nodes, or each node is directly connected with at least one node within the other communication group, which is not limited in the present application.

[0097] In the following, some possible implementation manners of the first communication connection and the second communication connection are exemplarily given taking specific application scenarios as examples.

[0098] Optionally, multiple nodes within the communication group 210 can be encapsulated in one module, and the intra-encapsulation communication has lower delay and higher bandwidth, which can realize higher computing performance and data transmission efficiency, and can save space and hardware resources. For example, each communication group 210 can include 2 nodes, the number of communication groups 210 is 4, and the total number of nodes is 8, so such a system can also be called an 8P system. The 2 nodes can be encapsulated in one module to form a chip with a double-die package architecture. The double-die package architecture allows two identical or different dies to be contained in one physical package. Here, the die can be an unencapsulated GPU, CPU, TPU, etc., so one die is one node, and one double-die package architecture constitutes one communication group 210.

[0099] In this scenario, the first communication connection can be a communication connection used between the double dies, that is, a communication mode that can be used by two dies encapsulated in the same chip, such as serial input / output (SIO), serial peripheral interface (SPI), inter-integrated circuit (I2C), etc., which is not limited in the present application.

[0100] The second communication connection can include a Huawei Cloud Communication Service (HCCS), an Nvlink communication connection, a switch chip connection, and the like. The HCCS is a high-speed interconnection technology for cloud computing scenarios, providing high-speed, low-latency intra-cloud communication services. The HCCS can enable direct connections between nodes while providing high bandwidth and reliability. The Nvlink communication connection is a high-speed interconnection technology for direct communication between GPUs, suitable for high-performance computing and machine learning. The switch chip is a chip with functions for transmitting electrical and / or optical signals and can transmit data based on rules, providing exclusive electrical or optical signal paths for any two nodes connected to the switch chip. If the switch chip is a two-layer switch (link layer switch), data transmission can be achieved through address resolution protocol (ARP) and other switch forwarding algorithms. If the switch chip is a three-layer switch (network layer switch), data transmission can be achieved through routing information protocol (RIP), border gateway protocol (BGP), and other routing algorithms. The present application is not limited in this regard.

[0101] Alternatively, the plurality of nodes within each communication group 210 can also be independent nodes that are each physically separate rather than integrated in the same package. For example, each communication group 210 can include 4 nodes, the number of communication groups 210 is 3, and the total number of nodes is 12, so such a system can also be referred to as a 12P system.

[0102] In this scenario, the first communication connection can include the HCCS, the Nvlink communication connection, the switch chip connection, and the like, and the second communication connection can include the HCCS, the Nvlink communication connection, the switch chip connection, and the like. The first communication connection and the second communication connection are different. For example, the first communication domain is implemented through the HCCS, and any two nodes within a communication group are directly connected through the HCCS in a full mesh. The second communication domain is implemented through the switch chip, and any two nodes of different communication groups are directly connected through the switch chip, or each node is directly connected to at least one node of a different communication group through the switch chip. The present application is not limited in this regard.

[0103] It should be noted that in the above examples, the training system includes two communication domains. In actual implementation, the training system can also include more than two communication domains. For example, in the example shown in FIG. 2, three communication groups can be regarded as a large group, and a third communication connection can be established between multiple large groups to form a third communication domain. Of course, the connection mode of the third communication connection is different from that of the first communication connection and the second communication connection. In this way, when synchronizing data, parallel transmission can be realized based on the three communication domains. The number of communication domains can be determined according to the actual business scenario, and the present application does not make specific limitations.

[0104] It should be noted that the training system shown in FIG. 2 can be a supernode in a distributed system used for AI model training. The distributed system can include multiple supernodes as shown in FIG. 2, and multiple supernodes in the distributed system can train the same model using different training data.

[0105] The structure of the training system provided by the present application is described above in combination with FIGS. 2 and 3, and in particular, how multiple nodes in the training system form multiple communication domains. In the following, how each node in the training system provided by the present application realizes collective communication is explained and described in combination with FIG. 4. FIG. 4 is a flowchart of collective communication of each node in a training system provided by the present application. FIG. 4 is described in detail with node 11 as an example. The steps of collective communication of other nodes can refer to node 11, and will not be described one by one. It should be noted that in the example shown in FIG. 4, the reduce-scatter algorithm is used for collective communication algorithm. Other collective communication algorithms can also be implemented in a similar way, which will not be described one by one.

[0106] First, when node 11 needs to perform collective communication with other nodes after the training system 200 starts training the model, node 11 can first obtain the bandwidth ratio and system structure information.

[0107] The bandwidth ratio refers to the ratio between the first bandwidth of the first communication domain and the second bandwidth of the second communication domain. The system structure information can include information related to the structure of the training system 200, such as the number of nodes, the number of communication groups, and the number of nodes in each communication group, etc. It can also include more information, such as the performance characteristics of the node, including the model of the node, the storage space, the training framework supported by the node, the hardware details, etc., which are not limited by the present application.

[0108] Secondly, the node 11 can determine the first communication amount of the first communication domain and the second communication amount of the second communication domain according to the system structure information and the bandwidth ratio, wherein the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, and the adaptation of the ratio and the bandwidth ratio can mean that the two are equal, that is, x=y; or the adaptation can mean that the two are approximate, that is, x≈y; or the adaptation can mean that the difference between the two is less than a threshold, that is, x-y≤threshold z. It should be understood that the adaptation means that the two are close or the difference is not enough to affect the collection communication efficiency, and the equal case is the best case, in which the bandwidth can be maximally utilized, and the closer the ratio is to the bandwidth ratio, the higher the bandwidth utilization rate is.

[0109] The first communication amount includes the number or fraction of sub-data pulled by the node 11 to the plurality of first communication nodes through the first communication domain, and the second communication amount includes the number or fraction of sub-data pulled by the node 11 to the plurality of second communication nodes through the second communication domain, wherein the data on each node is divided into a plurality of sub-data according to the total number of nodes, and each sub-data corresponds to a data number. The first communication node and the node 11 belong to the same communication group, and the second communication node and the node 11 belong to different communication groups.

[0110] For example, the first communication amount is 3, which means that the node 11 needs to pull a total of 3 sub-data from other nodes in the same group, and if the number of other nodes is 3, it means that each node pulls 1 sub-data. For another example, in the example of FIG. 1, the first communication node of node 0 is only node 1, and the first communication amount is 4, that is, a total of 4 sub-data is pulled from node 1. The second communication node of node 0 includes node 3, node 5 and node 7, and the second communication amount is 3, that is, a total of 3 sub-data is pulled from node 3, node 5 and node 7, and each node pulls 1 sub-data.

[0111] In a specific implementation, each node needs to pull sub-data from all other nodes, if the training system 200 has N communication groups, each communication group includes M nodes, and there are a total of MN nodes, and the data on each node is divided into MN sub-data, then the sum of the first communication amount and the second communication amount is MN-1 sub-data, and then based on the binary linear equation set, the first communication amount and the second communication amount can be calculated according to the ratio of the first communication amount and the second communication amount as the bandwidth ratio.

[0112] It should be understood that the first communication amount and the second communication amount are calculated based on the bandwidth ratio in the present application, so that the ratio of the communication amount is adapted to the bandwidth ratio, the communication amount of the communication domain with large bandwidth is increased, and the communication amount of the communication domain with small bandwidth is reduced, so that the bandwidth of the two communication domains can be maximally utilized, and the communication efficiency of the collection communication is improved.

[0113] Then, the node 11 can determine a first communication strategy of the first communication domain according to the first communication volume and a second communication strategy of the second communication domain according to the second communication volume.

[0114] The first communication strategy includes each first communication node in the first communication domain that communicates with the node 11 and first data pulled by the node 11 to each first communication node. The second communication strategy includes each second communication node in the second communication domain that communicates with the node 11 and second data pulled by the node 11 to each second communication node. The first communication volume can be obtained based on the first data and the number of first communication nodes, and the second communication volume can be obtained based on the second data and the number of second communication nodes.

[0115] Specifically, the first communication strategy includes a node number of each first communication node and a data number of the first data, and the second communication strategy includes a node number of each second communication node and a data number of the second data. It should be understood that each node in the training system 200 corresponds to a unique identifier, that is, a node number, which can be a number or a character number. The first data includes at least one sub-data, and the data number of the first data refers to the data number of the at least one first sub-data. The second data includes at least one sub-data, and the data number of the second data refers to the data number of the at least one second sub-data.

[0116] For example, in the example shown in FIG. 1, the node number of the first communication node of the node 0 is B, and the data number of the first data is 0, 2, 4, and 5. The node number of the second communication node is D, F, and H, and the data number of the second data is 0. The above example is used for illustration, and the present application is not limited in this regard.

[0117] Finally, the node 11 performs in-group communication using the first communication domain based on the first communication strategy and performs inter-group communication using the second communication domain based on the second communication strategy to complete the collective communication and obtain a collective communication result. Based on the collective communication result, the training of the model can be completed.

[0118] Specifically, based on the node number and the data number, the node 11 can obtain the first data from one or more first communication nodes through the first communication domain, and the node 22 can obtain the second data from one or more second communication nodes through the second communication domain. Since the above communication nodes and data are determined based on the first communication volume and the second communication volume, and the ratio between the first communication volume and the second communication volume is the bandwidth ratio between the first communication domain and the second communication domain, the bandwidth of the two communication domains can be maximally utilized, and the problem of bandwidth resource waste does not occur, thereby improving the efficiency of the collective communication.

[0119] For example, taking the structure of the training system shown in FIG. 1 as an example, the total number of nodes is 8, the number of communication groups is 4, the number of nodes in each communication group is 2, and if the bandwidth ratio is 1:6, according to the traditional technical solution shown in FIG. 1, the traffic ratio is 3:4, which causes the bandwidth of the second communication domain to be wasted, affecting the communication efficiency. According to the technical solution provided in the present application, node 0 can determine that the first communication traffic is 1 and the second communication traffic is 6, and then determine that the node number of the first communication node is B, the data number of the first data is 0, the node number of the second communication node is C-H, and the data number of the second data is 0. In this way, node 0 obtains data B0 from node 1 through the first communication domain, and obtains data C0-H0 from nodes 2-7 through the second communication domain, thereby completing the reducescatter set communication algorithm. In this way, the traffic ratio of the first communication domain and the second communication domain is 1:6, which is exactly the same as the bandwidth ratio, so that the bandwidth of the two communication domains can be maximally utilized, improving the communication efficiency.

[0120] Similarly, if the bandwidth ratio is 2:5, the communication strategy with a traffic ratio of 2:5 can also be obtained according to the technical solution of the present application, and so on. Here, the other structures of the training system 200 can also use a similar way to determine the first communication traffic and the second communication traffic based on the bandwidth ratio, and then determine the communication strategy of the two communication domains based on the traffic, so that the traffic ratio is adapted to the bandwidth ratio, and the bandwidth of the two communication domains can be maximally utilized, improving the communication efficiency.

[0121] Alternatively, node 11 can also pre-store the mapping relationship between different bandwidth ratios and different communication strategies under each system structure. In this way, when the bandwidth of the same structure fluctuates, node 11 can also determine the communication strategy corresponding to the fluctuated bandwidth ratio according to the mapping relationship, thereby improving the efficiency of obtaining the communication strategy and further improving the communication efficiency. The above mapping relationship can be pre-set by the system or continuously updated by node 11, that is, node 11 calculates the communication strategy according to the historical bandwidth ratio during the model training process, and then continuously updates the mapping relationship, which is not limited in the present application. In the specific implementation, when matching the mapping relationship, if the current bandwidth ratio obtained by node 11 is similar to the pre-set bandwidth ratio, the communication strategy corresponding to the pre-set bandwidth ratio can also be used as the communication strategy corresponding to the current bandwidth ratio. The threshold between the current bandwidth ratio and the pre-set bandwidth ratio is less than the threshold, which can be determined as similar.

[0122] For example, the node 11 pre-stores preset structure information corresponding to a plurality of possible system structures, and stores a plurality of possible preset bandwidth ratios corresponding to each possible system structure, and then stores a communication strategy corresponding to each preset bandwidth ratio. For example, the node 11 stores a system structure in which the number of nodes is 8, the number of communication groups is 4, and the number of nodes in each communication group is 2, and the preset bandwidth ratios that can exist in the system structure include 1:6, 2:5, 3:4, and 4:3, and each preset bandwidth ratio corresponds to a communication strategy, which includes the first communication strategy and the second communication strategy. In this way, if the node 11 obtains that the current bandwidth ratio is 2:5, the first and second communication strategies can be obtained according to the mapping relationship. If the node 11 obtains that the current bandwidth ratio is 3:5, the preset bandwidth ratio closest to the bandwidth ratio is 3:4, and at this time, the traffic ratio corresponding to the current bandwidth ratio 3:5 of the node 11 can be determined according to the preset bandwidth ratio 3:4 and the preset traffic ratio 3:4. The above example is used for illustration, and the present application is not limited in this way.

[0123] It should be noted that the node 11 is taken as an example to describe the process of collective communication in FIG. 4, and the operations of other nodes in collective communication are the same as that of the node 11, which will not be illustrated one by one.

[0124] It should be noted that the node 11 is taken as an example to describe the process of collective communication in FIG. 4, and the operations of other nodes in collective communication are the same as that of the node 11, which will not be illustrated one by one.

[0125] It should be noted that the node 11 is taken as an example to describe the process of collective communication in FIG. 4, and the operations of other nodes in collective communication are the same as that of the node 11, which will not be illustrated one by one.

[0126] The training system provided in the present application is explained and described above in combination with FIGS. 2-4, and the data communication method provided in the present application is explained and described below in combination with FIGS. 5-13, which can be applied to the training system 200 shown in FIGS. 2-4.

[0127] FIG. 5 is a schematic flow diagram of steps of a data communication method provided in the present application, as shown in FIG. 5, the method can include the following steps:

[0128] S510: The first node obtains bandwidth ratio and system structure information. Wherein, the first node can be the node 11 in FIGS. 2-4.

[0129] In a specific implementation, the bandwidth ratio refers to the ratio between the first bandwidth of the first communication domain and the second bandwidth of the second communication domain. The first bandwidth refers to the total bandwidth of the first communication domain, and the second bandwidth refers to the total bandwidth of the second communication domain. The first bandwidth and the second bandwidth can be obtained by a bandwidth detection tool. Specifically, the bandwidth detection tool can include a tool for testing bandwidth by sending and receiving data packets, a tool for determining bandwidth by analyzing network real-time data and historical data, a bandwidth detection tool built-in a router or a switch, and a bandwidth detection tool provided by an Internet service provider, which are not limited in the present application.

[0130] The system structure information can include information related to the structure of the training system 200, such as the total number of nodes, the number of communication groups, and the number of nodes in each communication group, etc., and can also include more information, such as the performance characteristics of the nodes, including the model number, storage space, training framework supported by the node, hardware details, etc., which are not limited in the present application.

[0131] It should be noted that the data communication method provided in the present application is used to implement a distributed training system, and to realize the collective communication between multiple nodes. Therefore, before S510, the training system 200 can receive the model training request sent by the client 100, and in response to the model training request, the training system 200 can start training the model, cut and distribute the training task of the model to each node for training. When each node needs to obtain the training result of other nodes, each node can act as a first node to execute S510-S550 to obtain the corresponding collective communication result.

[0132] Optionally, the model trained by the training system 200 can be an AI model, including but not limited to a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), a large language model (LLM), a transformer, etc., which are not limited in the present application.

[0133] Optionally, when the training system 200 distributes the training tasks cut into each node for training, the algorithm used for cutting includes data parallelism, tensor parallelism, pipeline parallelism, etc., wherein the data parallelism refers to distributing the dataset cut into multiple nodes for training, the tensor parallelism refers to distributing the model parameters into multiple nodes for training, and the pipeline parallelism refers to distributing the model cut according to the network layer into multiple nodes for training, and more algorithms can also be included, which are not listed one by one here.

[0134] S520: The first node determines the first communication amount and the second communication amount.

[0135] In specific implementations, the first communication amount includes the total amount of communication of the first node with other nodes in the same communication group in the first communication domain, and the second communication amount includes the total amount of communication of the first node with nodes in other communication groups in the second communication domain. The first node can determine the first communication amount and the second communication amount according to the bandwidth ratio and the system structure information obtained in S510, wherein the ratio between the first communication amount and the second communication amount is the same as the bandwidth ratio.

[0136] In specific implementations, the unit of the communication amount is the number or fraction of sub-data, wherein the data on each node is divided into rs parts according to the total number of nodes rs, such as the system structure shown in FIG. 1, the number of nodes rs = 8, and the data on each node is divided into 8 sub-data, node 0 includes sub-data A0-A7, node 1 includes sub-data B0-B7, and so on. If the first communication amount is 3, it means that the first node obtains 3 fractions of sub-data from other nodes in the first communication group, and the second communication amount is 4, which means that the first node obtains 4 fractions of sub-data from nodes in other communication groups. The above examples are used for illustration, and the present application is not limited specifically.

[0137] In a specific implementation, the first node needs to obtain data on each of the other nodes, so the total amount of the first traffic and the second traffic is fixed. If the number of nodes in a communication group is M, the number of communication groups is N, the total number of nodes is rs=MN, and the data on each node is divided into rs parts, then the total amount of the first traffic and the second traffic is MN-1. If the first traffic is pn and the second traffic is hn, then pn+hn=rs-1, and the ratio between the first bandwidth and the second bandwidth is R, the ratio between the first traffic and the second traffic is pn / hn=R, and then the values of the first traffic pn and the second traffic hn can be obtained based on the binary linear equation set.

[0138] For example, assuming that the system structure is as shown in FIG. 1, the number of communication groups N=4, the number of nodes in each communication group M=2, and the bandwidth ratio is 1 / 6, then pn+hn=7 and pn / hn=1 / 6. Based on this, pn=1 and hn=6 can be calculated. For another example, if the bandwidth ratio is 2 / 5, then pn+hn=7 and pn / hn=2 / 5. Based on this, pn=2 and hn=5 can be calculated. In this way, no matter what the bandwidth ratio is, the same first traffic and second traffic can be calculated, and the ratio between the first traffic and the second traffic is adapted to the bandwidth ratio.

[0139] It should be noted that, since the unit of the traffic is the number or fraction of sub-data, if the calculation result has a decimal, an integer operation can be performed, which is not illustrated here.

[0140] S530: The first node determines a first communication strategy of the first communication domain and a second communication strategy of the second communication domain.

[0141] First, the parameters contained in the first communication strategy and the second communication strategy are explained.

[0142] In the embodiment of the present application, the first node can determine the first communication strategy according to the first traffic. The first communication strategy includes information of a first communication node that needs to be communicated by the first node in the first communication domain, and information of first data that needs to be pulled by the first node from each first communication node. After the first node pulls the first data from each first communication node, the total amount of communication is the first traffic.

[0143] Similarly, the first node can determine the second communication strategy according to the second traffic. The second communication strategy includes information of a second communication node that needs to be communicated by the first node in the second communication domain, and information of second data that needs to be pulled by the first node from each second communication node. After the first node pulls the second data from each second communication node, the total amount of communication is the second traffic.

[0144] Optionally, the first data and the second data refer to data that needs to be communicated in a collective manner in the model training process, such as model weights, gradients, activation values, loss, model state information, hyperparameters, attention parameters, and the like, which are not limited in the present application.

[0145] In specific implementations, the information of the first communication node and the second communication node can be node numbers, such as node 0 and node 1 shown in FIG. 1. The information of the first data and the second data can be the numbers of the sub-data that needs to be pulled. Among them, the data on each node is divided into multiple sub-data according to the total number of nodes, and each sub-data has a corresponding number. For example, node 0 in FIG. 1 includes data A0-A7, and node 1 includes data B0-B7, which are the numbers of the sub-data.

[0146] For example, in the example shown in FIG. 1, the node number of the first communication node of node 0 is B, the data numbers of the first data are 0, 2, 4, and 5, the node numbers of the second communication node are D, F, and H, and the data numbers of the second data are 0. The above example is used for illustration, and the present application is not limited thereto.

[0147] Of course, the first communication strategy and the second communication strategy can include more content, which can be determined according to the actual business scenario, and the present application is not limited thereto.

[0148] Secondly, how to determine the parameters in the first communication strategy and the second communication strategy is explained.

[0149] In the embodiments of the present application, for the first communication strategy, the first communication node can be all nodes in the first communication domain except the first node, that is, the first node needs to communicate with all other nodes in the same communication group (that is, the first communication group), and therefore the number of the first communication node can be determined according to the communication group in which the first node is located.

[0150] For example, if the first communication group has 4 nodes, node 0-node 3, and the first node is node 0, then the number of the first communication node is 1, 2, and 3. For another example, if the first communication group has 2 nodes, node 0 and node 1, and the first node is node 0, then the number of the first communication node is 1. The above examples are used for illustration, and the present application is not limited thereto.

[0151] Further, the first data can include first sub-data, and the number of the first sub-data is the number of the sub-data required by the set communication result of the first node. For example, the set communication result of the first node is A0+B0+…, that is, the first sub-data on each node, and the number of the first sub-data is 0. If the set communication result of the first node is A5+B5+…, that is, the sixth sub-data on each node, and the number of the first sub-data is 5. Generally, if the node number and the sub-data number use the same coding mode, such as using numbers to represent the node number and the sub-data number, the node number of the first node and the number of the first sub-data are the same. For example, node 1 needs to obtain the sub-data numbered 1 on all nodes, and the number of the first sub-data is 1. Node 5 needs to obtain the sub-data numbered 5 on all nodes, and the number of the first sub-data is 5.

[0152] Further, the first data can further include one or more second sub-data, and the difference between the number of the second sub-data and the number of the first sub-data is the number M of nodes in the communication group or a multiple of M. For example, the number of the first sub-data is 0, and the number of nodes M in the communication group is 4, and then the number of the second sub-data can be 4, 8, 12, etc. The number of the second sub-data can be determined according to the first communication amount. If the first communication amount is pn, and the number of the first communication node is M-1, then the number of sub-data contained in each first data is sn=pn / (M-1), and the sum of the number of the first sub-data and the number of the second sub-data is sn, so the number of the second sub-data is sn-1.

[0153] It should be noted that the total number of nodes rs=MN, and the maximum number of sub-data on each node is MN. If the number of the second sub-data calculated according to the multiple of M is greater than MN, the excess part can be calculated from 0 onwards to obtain the actual number of the second sub-data. For example, the number of the second sub-data is calculated to be 9, but the total number of nodes is 8, and the excess part is 1. The actual number of the second sub-data can be obtained by calculating from 0 onwards, which is 1. For another example, the number of the second sub-data is calculated to be 15, but the total number of nodes is 12, and the excess part is 3. The actual number of the second sub-data can be obtained by calculating from 0 onwards, which is 3. The above examples are used for illustration, and the present application is not limited in this way.

[0154] For example, the determination method of the number of each sub-data in the first data is summarized in the formula as follows: formula (1) is obtained. j =(rank+M×j)%rs,0≤j≤sn-1 (1)

[0155] Wherein, sr jThe sr refers to the number of each sub-data in the first data, sr0=4 indicates that the data number of the first sub-data is 4, sr1=6 indicates that the data number of the second sub-data is 6, and so on. The rank refers to the node number of the first node, for example, rank=0 indicates that the first node is node 0, and rank=1 indicates that the first node is node 1. M is the number of nodes in the communication group, sn-1 is the number of the second sub-data, and rs is the total number of nodes of the training system 200. The symbol % indicates the modulus, that is, the remainder obtained by calculating (rank+M×j) divided by rs. By taking the modulus, when the number of the second sub-data calculated according to the multiple of M is greater than rs, the excess part can be calculated from 0 backward to obtain the actual number of the second sub-data. When it is not greater than rs, the remainder is rank+M×j, that is, the number of the first sub-data is rank, and the difference between the second sub-data and the first sub-data is a multiple of M.

[0156] It should be understood that the above formula (1) is an example of calculating the number of each sub-data in the first data provided by the present application, which is not limited by the present application. In addition, formula (1) is an example of formula under the scenario of reducescatter algorithm. For other collective communication algorithms, such as allgather, allreduce, and the like, other calculation formulas can be used to determine the data number of the first data, in combination with the collective communication result required by the communication algorithm and the first communication amount, to infer the number of sub-data required to be sent by each first communication node. Here, examples are not shown one by one.

[0157] In the embodiment of the present application, for the second communication strategy, the data number of the second data includes the number of the first sub-data. That is, in the second communication domain, the first node needs to obtain the first sub-data from each second communication node, so that the first node can obtain the final collective communication result in combination with the first sub-data obtained in the first communication domain. As known from the foregoing, the number of the first sub-data is the number of the sub-data required by the collective communication result of the first node, for example, the collective communication result of the first node is A0+B0+…, and the number of the first sub-data is 0. Therefore, in the second communication strategy, the data number of the second data is determined, that is, the number of the first sub-data, and mainly the node number of the second communication node needs to be calculated.

[0158] Further, the second communication node can include a full communication node and a single communication node, wherein the full communication node refers to all nodes in a whole communication group needing to communicate with the first node, and the single communication node refers to only one node in a whole communication group needing to communicate with the first node. The total communication amount of the full communication node and the single communication node is determined according to the second communication amount, the total number of nodes of the full communication node and the single communication node is determined according to the number N of the communication groups, and the number h1 of the full communication nodes and the number h2 of the single communication nodes are determined based on the total communication amount and the total number of nodes. In this way, if the number of nodes in a communication group is M, the second communication amount is hn (the first communication amount pn and the second communication amount hn are calculated in step S520), the number of full communication nodes is h1, and the number of single communication nodes is h2, then the second communication amount hn = h1 + h2. In combination with the number N of the communication groups, the first node needs to communicate with each of the other communication groups, and h1 / M + h2 = N-1 can be determined. Based on the binary linear equation set, the number h1 of the full communication nodes and the number h2 of the single communication nodes can be calculated.

[0159] Further, when determining the number of the second communication node, the number h1 of the full communication nodes and the number h2 of the single communication nodes can be determined first, then the first communication group in which the first node is located can be determined, then h1 nodes after the first communication group are determined as the full communication nodes, and the node number of each full communication node can be determined according to the numbering order.

[0160] Further, after the node number of the full communication node is determined, h2 single communication nodes are determined from the remaining h2 communication groups, one single communication node is determined from one communication group, and the numbering order position of the single communication node of each communication group in the nodes in the communication group is the same as the numbering order position of the first node in the first communication group.

[0161] The numbering order position refers to the order position of the node in the communication group after ordering by number size. Assuming that the first node is the i th node of the first communication group, the full communication node is also the i th node in the communication group. For example, the total number of nodes is 12, the number of communication groups is 3, each communication group includes 4 nodes, the first communication group includes nodes 0, 1, 2, and 3, the second communication group includes nodes 4, 5, 6, and 7, and the third communication group includes nodes 8, 9, 10, and 11. Therefore, the numbering order positions of nodes 0, 4, and 8 are the same, and they are all the first nodes in the communication group in terms of numbering order. The numbering order positions of nodes 1, 5, and 9 are the same, and they are all the second nodes in the communication group in terms of numbering order. The numbering order positions of nodes 2, 6, and 10 are the same, and they are all the third nodes in the communication group in terms of numbering order. The numbering order positions of nodes 3, 7, and 11 are the same, and they are all the fourth nodes in the communication group in terms of numbering order. Therefore, if the first node is node 0, only nodes with numbering 1, 5, and 9 have the same numbering order position as node 0, and only these three nodes can be single communication nodes. The above example is used for illustration, and the present application is not limited in this regard.

[0162] For example, if the total number of nodes is 12, the number of communication groups N is 3, the number of nodes in each communication group M is 4, and the bandwidth ratio is 6 / 5, the first communication quantity pn is 6 and the second communication quantity hn is 5 according to the description of S520. According to h1 / M+h2=N-1 and hn=h1+h2, the number of full communication nodes h1 is 4 and the number of single communication nodes h2 is 1.

[0163] Assuming that the first node is node 0, the communication group 1 in which node 0 is located includes nodes 0-3, and the number of full communication nodes h1 is 4. Therefore, the four nodes after node 3, i.e., nodes 4, 5, 6, and 7, can be determined as full communication nodes, i.e., all nodes in communication group 2. Then, from the remaining one communication group, h2=1 single communication node is determined. The position of the single communication node in communication group 3 is the same as the position of node 0 in communication group 1, so the single communication node is the first node in communication group 3, i.e., node 8.

[0164] Therefore, the second communication nodes that node 0 needs to communicate in the second communication domain include nodes 4, 5, 6, 7, and 8, and the data number of the data is 0 (because the first node is node 0, node 0 needs to obtain the data with number 0 on each node). The above example is used for illustration, and the present application is not limited in this regard.

[0165] For example, the determination method of the number of each second communication node is summarized in the formula as follows: hr j =(rank+j)%rs,1≤j≤h1 (2) hrj = (rank - M x (j - h1) + rs) % rs, h1 < j ≤ h1 + h2

[0166] wherein, hr j It refers to the node number of the second communication node, hr1 = 1 indicates that the number of the first second communication node is 1, hr2 = 5 indicates that the number of the second second communication node is 5, and so on. rank refers to the number of the last node in the communication group where the first node is located, M is the number of nodes in the communication group, and rs is the total number of nodes of the training system 200. The first row formula in formula (2) is used to determine the number of h1 full communication nodes, and the second row formula is used to determine the number of h2 single communication nodes.

[0167] It should be understood that the above formula (2) is an example of calculating the node number of the second communication node provided by the present application, and the present application does not limit this, and formula (2) is an example of formula in the reducescatter algorithm scenario. For other collective communication algorithms, such as allgather, allreduce, and the like, other calculation formulas can also be used to determine the node number of the second communication node, in combination with the collective communication result required by the communication algorithm and the second communication amount, the node number of each second communication node can be inferred, which is not illustrated one by one here.

[0168] Briefly speaking, in the first communication domain, the node number of the first communication node is determined, that is, the number of all nodes in the first communication group except the first node, but the data number of the first communication data is not determined and needs to be calculated in combination with the first communication amount, such as the above formula (1) which is a possible calculation method. In the second communication domain, the data number of the second data is determined, that is, the number of the first group of data, but the number of the second communication node is not determined and needs to be calculated in combination with the second communication amount, such as the above formula (3) which is a possible calculation method.

[0169] It should be understood that each node can be the first node, and its first communication strategy and second communication strategy can be determined in the manner described above, which is not illustrated one by one here.

[0170] In order to enable the present application to be better understood, the communication strategies in the data communication method provided by the present application will be illustrated in combination with specific application scenarios in combination with FIGS. 6-13. Among them, FIGS. 6-9 take a system architecture as an example to illustrate the communication strategies under different bandwidth ratios. FIGS. 10-12 take another system architecture as an example to illustrate the communication strategies under different bandwidth ratios.

[0171] FIG. 6 is an example diagram of a communication strategy of a data communication method provided by the present application in a first application scenario, as shown in FIG. 6, in the application scenario, the set communication algorithm uses the reducescatter algorithm, the total number of nodes rs=8, the number of communication groups N=4, the number of nodes in the communication group M=2, and the bandwidth ratio R=1 / 6. The following describes the communication strategy determination process in this scenario.

[0172] First, based on the description of S520, the first communication quantity pn and the second communication quantity hn are calculated, according to pn+hn=rs-1 and pn / hn=R, it can be obtained that pn+hn=7 and pn / hn=1 / 6, and the total communication quantity of the first communication domain pn=1 and the total communication quantity of the second communication domain hn=6 can be obtained.

[0173] Next, based on the description of S530, the first communication strategy and the second communication strategy are calculated.

[0174] Suppose the first node is node 1, for the first communication strategy, the first communication node is all other nodes in the communication group where node 1 is located, that is, node 0. The number of sub-data included in each first data is sn= pn / (M-1)=1. According to formula (1), the number of sub-data in the first data is sr j =(1+2×j)%8, 0≤j≤sn-1, that is, j has only one value 0, so the first communication node has only one sub-data, the number of which is sr0=1.

[0175] Therefore, the first communication strategy includes: node 1 pulling the sub-data numbered 1 from node 0, that is, A1.

[0176] For the second communication strategy, the second data only includes the first sub-data, that is, the data required by the set communication result of the first node, and the set communication result of node 1 is A1+B1+…+H1, so the number of the first sub-data of node 1 is 1, and the number of the sub-data of the second data is 1. The second communication node includes h1 full communication nodes and h2 single communication nodes, according to the second communication quantity hn=h1+h2. h1 / M+h2=N-1, it can be obtained that 6=h1+h2. h1 / 2+h2=3, it is calculated that h1=6 and h2=0, that is, the second communication node is all full communication nodes, according to formula (2), the node number of the second communication node includes: hr1=(1+1)%8=2, hr2=(1+2)%8=3, hr3=(1+3)%8=4, hr4=(1+4)%8=5, hr5=(1+5)%8=6, and hr6=(1+6)%8=7.

[0177] Therefore, the second communication strategy includes that the node 1 pulls the sub-data numbered 1, i.e., C1, D1, E1, F1, G1 and H1, from the nodes 2, 3, 4, 5, 6 and 7.

[0178] In this way, the node 1 obtains the data A1 based on the first communication domain by using the first communication strategy, obtains the data C1-H1 based on the second communication domain by using the second communication strategy, and obtains the final reducescatter communication result A1+B1+…+H1 by combining B1 locally of the node 1. When the set communication is performed, the ratio of the communication amount between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, which is 1:6, so that the bandwidth utilization of the communication domain is maximized, and the problem of bandwidth resource waste is avoided.

[0179] Similarly, other nodes can also determine their communication strategies in a similar way. For example, in FIG. 6, the solid arrows show the communication strategy of the node 1, and the dashed arrows show the communication strategy of the node 0, that is, in the first communication domain, the node 0 pulls the sub-data B0 of the node 1, and in the second communication domain, the node 0 pulls the sub-data numbered 0 from the nodes 2-7, so as to obtain the communication result A0+B0+…+H0. In FIG. 6, the data number that each node needs to pull in the group is represented by a dark rectangular block, for example, the node 2 needs to pull the sub-data numbered 2 from the node 3, and the node 3 needs to pull the sub-data numbered 3. Here, the details are not repeated.

[0180] FIG. 7 is a communication strategy example diagram of a data communication method provided by the present application in a second application scenario. The system structure of the application scenario is the same as that of FIG. 6, and the only difference is that the bandwidth ratio R=2 / 5. The communication strategy determination process in this scenario is described below in combination with the description above.

[0181] First, the first communication amount pn and the second communication amount hn are calculated based on the description of S520. According to pn+hn=rs-1 and pn / hn=R, pn+hn=7 and pn / hn=2 / 5 can be obtained, and the total communication amount pn of the first communication domain is 2 and the total communication amount hn of the second communication domain is 5.

[0182] Then, the first communication strategy and the second communication strategy are calculated based on the description of S530.

[0183] Suppose the first node is the node 1. For the first communication strategy, the first communication node is all other nodes of the communication group to which the node 1 belongs, i.e., the node 0. The number of sub-data included in each first data is sn=pn / (M-1)=2. According to formula (1), the number of sub-data in the first data is sr j=(1+2xj)%8,0≤j≤sn-1, that is, the value of j includes 0 and 1, so the first communication node includes 2 sub-data, and the numbers are sr0=1 and sr1=3.

[0184] Therefore, the first communication strategy includes that the node 1 pulls the sub-data numbered 1 and 3, that is, A1 and A3, from the node 0.

[0185] For the second communication strategy, based on the description of FIG. 6, the number of the sub-data of the second data is 1. The second communication node includes h1 full communication nodes and h2 single communication nodes, and according to the second communication quantity hn=h1+h2, h1 / M+h2=N-1, 5=h1+h2 can be obtained. h1 / 2+h2=3, and the number of full communication nodes h1=4 and the number of single communication nodes h2=1 are calculated, and according to formula (2), the node numbers of the full communication nodes include hr1=(1+1)%8=2, hr2=(1+2)%8=3, hr3=(1+3)%8=4, and hr4=(1+4)%8=5, and the node numbers of the single communication nodes include 7.

[0186] Therefore, the second communication strategy includes that the node 1 pulls the sub-data numbered 1 from the nodes 2, 3, 4, 5 and 7, wherein the node 7 pulls the sub-data G1 in the node 6 through the first communication domain, so when the node 1 pulls the sub-data numbered 1 from the node 7, H1 and G1 will actually be pulled, and therefore the sub-data pulled by the node 1 through the second communication strategy includes C1-H1.

[0187] In this way, the node 1 obtains the data A1 based on the first communication domain and using the first communication strategy, obtains the data C1-H1 based on the second communication domain and using the second communication strategy, and obtains the final reducescatter communication result A1+B1+…+H1 by combining B1 locally of the node 1, and when the set communication is performed, the ratio of the communication quantities between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, that is, 2:5, so that the bandwidth utilization of the communication domain is maximized, and the problem of bandwidth resource waste is avoided.

[0188] Similarly, other nodes can also determine their communication strategies in a similar manner, for example, in FIG. 1, the communication strategy of the node 0 is shown by a dashed arrow, and the numbers of the data pulled by each node in the group are also exemplarily shown by dark rectangular blocks, for example, the node 2 needs to pull the sub-data numbered 2 and numbered 4 from the node 3, and the node 3 needs to pull the sub-data numbered 3 and numbered 5, which will not be repeated and described herein.

[0189] FIG. 8 is an example diagram of a communication strategy of a data communication method provided by the present application in a third application scenario, the system structure of the third application scenario is the same as that of FIG. 6 and FIG. 7, the only difference is that the bandwidth ratio R = 2 / 5. The communication strategy determination process in the third application scenario is described below in combination with the above description. Since the calculation process is similar to that of FIG. 6 and FIG. 7, the process is described briefly here, and the detailed description can be referred to the description of FIG. 6 and FIG. 7.

[0190] First, according to pn+hn=7 and pn / hn=3 / 4, the total communication amount of the first communication domain pn=3 and the total communication amount of the second communication domain hn=4 can be obtained.

[0191] Then, the first communication strategy and the second communication strategy are calculated based on the description of S530.

[0192] Suppose the first node is node 1. For the first communication strategy, the first communication node is node 0. The number of sub-data included in each first data is sn=pn / (M-1)=3. According to formula (1), the numbers of sub-data in the first data are sr0=1, sr1=3, and sr2=5.

[0193] Therefore, the first communication strategy includes that node 1 pulls the sub-data with numbers 1, 3, and 5, i.e., A1, A3, and A5, from node 0.

[0194] For the second communication strategy, the number of sub-data of the second data is 1. According to 4=h1+h2 and h1 / 2+h2=3, the total communication node number h1=2 and the single communication node number h2=2 are calculated. According to formula (2), the node numbers of the total communication nodes include hr1=(1+1)%8=2 and hr2=(1+2)%8=3, and the node numbers of the single communication nodes include 5 and 7.

[0195] Therefore, the second communication strategy includes that node 1 pulls the sub-data with number 1 from nodes 2, 3, 5, and 7. Node 7 will pull the sub-data G1 in node 6 through the first communication domain, and node 5 will pull the sub-data E1 in node 4 through the first communication domain. Therefore, when node 1 pulls the sub-data with number 1 from node 5, it will actually pull E1 and F1, and when node 1 pulls the sub-data with number 1 from node 7, it will actually pull H1 and G1. Therefore, the sub-data pulled by node 1 through the second communication strategy includes C1-H1.

[0196] In this way, the node 1 obtains the data A1 based on the first communication domain using the first communication strategy, obtains the data C1-H1 based on the second communication domain using the second communication strategy, and obtains the final reducescatter communication result A1+B1+…+H1 in combination with B1 locally of the node 1. When the communication is performed in a set manner, the ratio of the communication amount between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, which is 3:4, so that the bandwidth utilization of the communication domain can be maximized, and the problem of bandwidth resource waste can be avoided.

[0197] Similarly, other nodes can also determine the communication strategy in a similar manner. For example, in FIG. 1, the communication strategy of the node 0 is shown by the dashed arrow, and the data numbers that each node needs to pull in the group are also exemplarily shown by the dark rectangular blocks. For example, the node 2 needs to pull the sub-data with numbers 2, 4, and 6 from the node 3, and the node 3 needs to pull the sub-data with numbers 3, 5, and 7. Details are not repeated here.

[0198] FIG. 9 is a communication strategy example of a data communication method provided by the present application in a fourth application scenario. The system structure in this application scenario is the same as that in FIGS. 6-8, and the only difference is that the bandwidth ratio R=4 / 3. The communication strategy determination process in this scenario is exemplarily described below in combination with the foregoing description. Since the calculation process is similar to that in FIGS. 6-8, the process is briefly described here, and the detailed description can be referred to the related description in FIGS. 6 and 7.

[0199] First, according to pn+hn=7 and pn / hn=4 / 3, the total communication amount pn of the first communication domain is 4, and the total communication amount hn of the second communication domain is 3.

[0200] Then, the first communication strategy and the second communication strategy are calculated based on the description of S530.

[0201] Suppose the first node is the node 1. For the first communication strategy, the first communication node is the node 0. The number of sub-data included in each first data is sn=pn / (M-1)=4. According to formula (1), the numbers of the sub-data in the first data are sr0=1, sr1=3, sr2=5, and sr2=7.

[0202] Therefore, the first communication strategy includes that the node 1 pulls the sub-data with numbers 1, 3, 5, and 7, i.e., A1, A3, A5, and A7, from the node 0.

[0203] For the second communication strategy, the number of the sub-data of the second data is 1. According to 3=h1+h2 and h1 / 2+h2=3, the number of the full communication nodes h1 is 0, and the number of the single communication nodes h2 is 3. According to formula (2), the node numbers of the single communication nodes are 3, 5, and 7.

[0204] Therefore, the second communication strategy includes that node 1 pulls the sub-data numbered 1 from nodes 3, 5 and 7, wherein node 3 pulls the sub-data C1 in node 2 through the first communication domain, node 7 pulls the sub-data G1 in node 6 through the first communication domain, and node 5 pulls the sub-data E1 in node 4 through the first communication domain, so that when node 1 pulls the sub-data numbered 1 from node 3, it actually pulls C1 and D1, when node 1 pulls the sub-data numbered 1 from node 5, it actually pulls E1 and F1, and when node 1 pulls the sub-data numbered 1 from node 7, it actually pulls H1 and G1, so that the sub-data pulled by node 1 through the second communication strategy includes C1 to H1.

[0205] In this way, node 1 obtains the data A1 based on the first communication domain using the first communication strategy, obtains the data C1 to H1 based on the second communication domain using the second communication strategy, and obtains the final reducescatter communication result A1+B1+…+H1 in combination with B1 locally stored in node 1, and the ratio of the communication amount between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, which is 4:3, so that the bandwidth utilization of the communication domain is maximized, and the problem of bandwidth resource waste is avoided.

[0206] Similarly, other nodes can also determine their communication strategies in a similar manner. For example, in FIG. 1, the communication strategy of node 0 is shown by a dashed arrow, and the data numbered to be pulled by each node in the group is also exemplarily shown by a dark rectangular block, for example, node 2 needs to pull the data numbered 0, 2, 4 and 6 from node 3, and node 3 needs to pull the sub-data numbered 1, 3, 5 and 7, which will not be repeated and described herein.

[0207] It should be understood that FIGS. 6 to 9 exemplarily show the bandwidth ratio that can occur in the training system 200 under this structure, and the example of the corresponding communication strategy. The bandwidth ratio that can occur under another structure and the corresponding communication strategy will be exemplarily described below in combination with FIGS. 10 to 12.

[0208] FIG. 10 is a communication strategy example diagram of a data communication method provided by the present application under a fifth application scenario, as shown in FIG. 10, under this application scenario, the collection communication algorithm uses the reducescatter algorithm, the total number of nodes rs=12, the number of communication groups N=3, the number of nodes in the communication group M=4, and the bandwidth ratio R=9 / 2. The communication strategy determination process under this scenario will be exemplarily described below in combination with the description above.

[0209] First, based on the description of S520, the first communication quantity pn and the second communication quantity hn are calculated, according to pn+hn=rs-1 and pn / hn=R, it can be obtained that pn+hn=11 and pn / hn=9 / 2, the total communication quantity of the first communication domain pn=9 and the total communication quantity of the second communication domain hn=2 can be obtained.

[0210] Next, based on the description of S530, the first communication strategy and the second communication strategy are calculated.

[0211] Suppose the first node is node 0, for the first communication strategy, the first communication node is all the other nodes in the communication group where node 0 is, that is, nodes 1, 2 and 3. The number of sub-data included in each first data is sn=pn / (M-1)=3. According to formula (1), the number of sub-data in the first data is sr=(0+4×j)mod 12, 0≤j≤sn-1, that is, j can be 0, 1 and 2, based on which the number of sub-data in the first communication node includes 0, 4 and 8. j

[0212] Therefore, the first communication strategy includes that node 0 pulls the sub-data with the number of 0, 4 and 8 from nodes 1, 2 and 3, that is, B0, B4 and B8 from node 1, C0, C4 and C8 from node 2, and D0, D4 and D8 from node 3.

[0213] For the second communication strategy, the second data only includes the first sub-data, that is, the data required by the set communication result of the first node, the set communication result of node 0 is A0+B0+…+H0, so the number of the first sub-data of node 0 is 0, and the number of the sub-data of the second data is 0. The second communication node includes h1 full communication nodes and h2 single communication nodes, according to the second communication quantity hn=h1+h2. h1 / M+h2=N-1, it can be obtained that 2=h1+h2. h1 / 4+h2=2, it is calculated that h1=0 and h2=2, that is, all the second communication nodes are single communication nodes, according to formula (2), the node number of the second communication node includes 4 and 8.

[0214] Therefore, the second communication strategy includes that node 0 pulls the sub-data with the number of 0 from nodes 4 and 8, since node 4 pulls F0, G0 and H0 from nodes 5, 6 and 7 through the first communication domain, node 0 pulls the sub-data with the number of 0 from node 4, which actually pulls E0~H0. Since node 8 pulls J0, K0 and L0 from nodes 9, 10 and 11 through the first communication domain, node 0 pulls the sub-data with the number of 0 from node 8, which actually pulls I0~L0.

[0215] ​In this way, the node 0 obtains the data B0-D0 based on the first communication domain and using the first communication strategy, and obtains the data E0-L0 based on the second communication domain and using the second communication strategy, and combines A0 of the node 0 to obtain the final reducescatter communication result A0+B0+…+L0. When the set communication is performed, the ratio of the communication amount between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, which is 9:2, so that the bandwidth utilization of the communication domain is maximized, and the problem of bandwidth resource waste is avoided.

[0216] Similarly, other nodes can also determine their communication strategies in a similar manner. For example, in FIG. 1, the dark rectangular blocks on each node represent the numbers of the first data, i.e., the numbers of the sub-data that the node needs to pull from other nodes in the same communication group. For example, the node 1 needs to pull the sub-data numbered 1, 5, and 9 from the node 0, the node 2, and the node 3, and the node 2 needs to pull the sub-data numbered 2, 6, and 10 from the node 0, the node 1, and the node 3. Details are not described herein.

[0217] FIG. 11 is a communication strategy example of a data communication method provided in the present application in a sixth application scenario. The system structure in this application scenario is the same as that in FIG. 10, and the only difference is that the bandwidth ratio R = 6 / 5. The communication strategy determination process in this scenario is described below in combination with the above description.

[0218] First, the first communication amount pn and the second communication amount hn are calculated based on the description of S520. According to pn+hn = rs-1 and pn / hn = R, it can be obtained that pn+hn = 11 and pn / hn = 6 / 5, and the total communication amount of the first communication domain pn = 6 and the total communication amount of the second communication domain hn = 5.

[0219] Then, the first communication strategy and the second communication strategy are calculated based on the description of S530.

[0220] Suppose the first node is the node 0. For the first communication strategy, the first communication nodes are all the other nodes in the communication group where the node 0 is located, i.e., the node 1, the node 2, and the node 3. The number of sub-data included in each first data is sn = pn / (M-1) = 2. According to formula (1), the numbers of the sub-data in the first data are sr j =(0+4×j) % 12, 0≤j≤sn-1, that is, j can be 0 or 1, based on which the numbers of the sub-data in the first communication nodes include 0 and 4.

[0221] Therefore, the first communication strategy includes that the node 0 pulls the sub-data numbered 0 and 4 from the node 1, the node 2, and the node 3, i.e., B0 and B4 from the node 1, C0 and C4 from the node 2, and D0 and D4 from the node 3.

[0222] For the second communication strategy, the second data only includes the first sub-data, that is, the data required by the collective communication result of the first node, the collective communication result of node 0 is A0+B0+…+H0, so the first sub-data of node 0 is numbered 0, and the sub-data number of the second data is 0. The second communication node includes h1 full communication nodes and h2 single communication nodes, and according to the second communication amount hn=h1+h2. h1 / M+h2=N-1, 5=h1+h2 can be obtained. h1 / 4+h2=2, h1=4 and h2=1 are calculated, that is, the second communication node includes 4 full communication nodes and 1 single communication node, and according to formula (2), the node numbers of the 4 full communication nodes are 4, 5, 6 and 7, and the node number of the 1 single communication node is 8.

[0223] Therefore, the second communication strategy includes that node 0 pulls the sub-data numbered 0 from nodes 4, 5, 6 and 7 to obtain E0-H0, and pulls the sub-data numbered 0 from node 8 to obtain I0-L0, because node 8 pulls F0, G0 and H0 from nodes 9, 10 and 11 through the first communication domain, so node 0 pulls the sub-data numbered 0 from node 8, and actually pulls I0-L0.

[0224] In this way, node 0 obtains data B0-D0 based on the first communication domain using the first communication strategy, obtains data E0-L0 based on the second communication domain using the second communication strategy, and obtains the final reducescatter communication result A0+B0+…+L0 in combination with A0 locally of node 0. When collective communication is performed, the ratio of the communication amount between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, both of which are 6:5, so that the bandwidth utilization of the communication domain is maximized, and the problem of bandwidth resource waste is avoided.

[0225] Similarly, other nodes can also determine their communication strategies in a similar manner. For example, in FIG. 1, the dark rectangular blocks on each node represent the numbering of the first data, that is, the numbering of the sub-data that needs to be pulled from other nodes in the same communication group, for example, node 1 needs to pull sub-data numbered 1 and 5 from nodes 0, 2 and 3, and node 2 needs to pull sub-data numbered 2 and 6 from nodes 0, 1 and 3. This will not be described in detail.

[0226] FIG. 12 is a communication strategy example diagram of a data communication method provided by the present application in a seventh application scenario. The system structure of the application scenario is the same as that of FIGS. 10 and 11, and the only difference is that the bandwidth ratio R=3 / 8. The communication strategy determination process in this scenario is described below in combination with the above description.

[0227] First, based on the description of S520, the first communication quantity pn and the second communication quantity hn are calculated, according to pn+hn=rs-1, pn / hn=R, pn+hn=11, pn / hn=3 / 8 can be obtained, the total communication quantity of the first communication domain pn=3, the total communication quantity of the second communication domain hn=8 can be obtained.

[0228] Next, based on the description of S530, the first communication strategy and the second communication strategy are calculated.

[0229] Suppose the first node is node 0, for the first communication strategy, the first communication node is all the other nodes of the communication group where node 0 is, that is, nodes 1, 2, 3. The number of sub-data included in each first data is sn=pn / (M-1)=1. According to formula (1), the number of sub-data in the first data is sr=(0+4×j) % 12, 0≤j≤sn-1, that is, j can be 0, based on which the number of sub-data in the first communication node includes 0. j

[0230] Therefore, the first communication strategy includes: node 0 pulls the sub-data numbered 0 from nodes 1, 2 and 3, that is, B0 from node 1, C0 from node 2, and D0 from node 3.

[0231] For the second communication strategy, the second data only includes the first sub-data, that is, the data required by the set communication result of the first node, the set communication result of node 0 is A0+B0+…+H0, so the number of the first sub-data of node 0 is 0, and the number of the sub-data of the second data is 0. The second communication node includes h1 full communication nodes and h2 single communication nodes, according to the second communication quantity hn=h1+h2. h1 / M+h2=N-1, 8=h1+h2 can be obtained. h1 / 4+h2=2, h1=8, h2=0 can be calculated, that is, the second communication node includes 9 full communication nodes, according to formula (2), the node numbers of the 8 full communication nodes are 4, 5, 6, 7, 8, 9, 10, 11.

[0232] Therefore, the second communication strategy includes: node 0 pulls the sub-data numbered 0 from nodes 4, 5, 6, 7, 8, 9, 10, 11 to obtain E0~L0.

[0233] ​In this way, node 0 obtains data B0-D0 based on the first communication domain using the first communication strategy, obtains data E0-L0 based on the second communication domain using the second communication strategy, and obtains the final reducescatter communication result A0+B0+…+L0 in combination with A0 locally of node 0. When performing set communication, the ratio of the communication amount between the first communication domain and the second communication domain is the same as the ratio of the total bandwidth, which is 3:8, so that the bandwidth utilization of the communication domain can be maximized, and the problem of bandwidth resource waste can be avoided.

[0234] Similarly, other nodes can also determine their communication strategies in a similar manner. For example, in FIG. 1, a dark rectangular block on each node represents the number of first data, that is, the number of sub-data that needs to be pulled from other nodes in the same communication group. For example, node 1 needs to pull sub-data numbered 1 from node 0, node 2, and node 3, and node 2 needs to pull sub-data numbered 2 from node 0, node 1, and node 3. Details are not repeated here.

[0235] It should be understood that the above describes the communication strategies of the system structure with a total number of nodes of 8 under different bandwidth ratios in combination with FIGS. 6-9, and the communication strategies of the system structure with a total number of nodes of 12 under different bandwidth ratios in combination with FIGS. 10-12. In a specific implementation, there can be more types of system structures, such as a total number of nodes of 16, a total number of nodes of 24, and the like. Other system structures can also use a similar manner to determine the first communication amount and the second communication amount based on the bandwidth ratio and the system structure information, and then determine the first communication strategy based on the first communication amount and determine the second communication strategy based on the second communication amount. Details are not repeated here.

[0236] Optionally, after the first node calculates the corresponding communication strategies of each system structure under different bandwidth ratios based on S520 and S530, the first node can establish a mapping relationship between the system structure, the bandwidth ratio, and the communication strategy. For example, the communication strategies corresponding to the four possible bandwidth ratios under the system structure shown in FIGS. 6-9 are recorded, and the communication strategies corresponding to the three possible bandwidth ratios under the system structure shown in FIGS. 10-12 are recorded. In this way, if the current bandwidth ratio of the first node is constantly changing, the corresponding communication strategy can be constantly updated according to the mapping relationship, the repeated calculation is reduced, and the efficiency of obtaining the communication strategy is improved.

[0237] Briefly, after the current bandwidth ratio and the system structure information are obtained in S510, it is determined whether there is a corresponding communication strategy according to the mapping relationship. If there is, the communication strategy determined by the mapping relationship is directly used to perform S540A-S550. If there is not, the corresponding communication strategy is calculated in S520 and S530, and the mapping relationship is updated.

[0238] It should be noted that when matching the mapping relationship, a threshold can be set. If the difference between the current bandwidth ratio and a certain bandwidth ratio in the mapping relationship is less than the threshold, the communication strategy corresponding to the bandwidth ratio can be taken as the communication strategy corresponding to the current bandwidth ratio, that is, the bandwidth ratio in the mapping relationship that is closest to the current bandwidth ratio is found. If the difference between the current bandwidth ratio and each bandwidth ratio in the mapping relationship is not less than the threshold, or the current system structure does not exist in the mapping relationship, S520 and S530 can be executed to complete the corresponding calculation.

[0239] S540A: The first node completes the intra-group communication through the first communication domain based on the first communication strategy.

[0240] S540B: The first node completes the inter-group communication through the second communication domain based on the second communication strategy.

[0241] In a specific implementation, the execution time periods of S540A and S540B overlap, that is, S540A and S540B can be executed in parallel to complete the collective communication. When the first node performs the intra-group communication in the first communication domain, the first node obtains the first data from each first communication node according to the node number of the first communication node and the data number of the first data in the first communication strategy. When the first node performs the inter-group communication in the second communication domain, the first node obtains the second data from each second communication node according to the node number of the second communication node and the data number of the second data in the second communication strategy.

[0242] Further, the first node can complete the intra-group communication and the inter-group communication through the segmented transmission. The segmented transmission refers to a process of splitting data into multiple data segments and transmitting the data segments gradually. Each data segment can be a sub-data or part of a sub-data, which is not limited in the present application.

[0243] It should be understood that according to the foregoing description, the second communication node includes a single communication node, that is, only one single communication node in the entire communication group communicates with the first node, because the single communication node performs the intra-group communication and pulls the data on other nodes in the group. Therefore, when the inter-group communication is performed, the first node only needs to pull the data of the communication group from the single communication node and does not need to communicate with other nodes in the group. In this case, if the segmented communication is not used, the first node needs to wait until the single communication node completes the intra-group communication before starting the inter-group communication, which affects the communication efficiency. Through the segmented communication, the single communication node can send the data to the first node through the second communication domain while pulling the data on other nodes through the first communication domain, that is, the single communication node sends and receives at the same time, which saves the communication time and avoids the problem of communication waiting.

[0244] For example, taking FIG. 11 as an example, when node 1 pulls the data of node 8, node 8 is also constantly pulling the data of nodes 9, 10 and 11, so that the data is transmitted in segments, node 1 can first pull I0 on node 8, and then node 8 pulls J0 of node 9, after I0 is transmitted, node 1 can continue to pull J0 on node 8, and node 8 pulls K0 of node 10, and so on, so as to avoid node 1 waiting for the communication in node 8 group to be completed before pulling the data, thereby improving the communication efficiency. The above example is for life, and the present application is not limited.

[0245] S550: The first node obtains a set communication result based on the communication results of the intra-group communication and the inter-group communication, and performs distributed training of the AI model based on the set communication result.

[0246] In specific implementation, the first node can aggregate, synchronize and update the parameters of the model according to the set communication result, thereby realizing optimization and improvement of the model. In the subsequent training process of the model, if set communication between nodes is still needed, each node can serve as the first node, obtain the corresponding communication strategy according to the description of S510-S550, complete set communication according to the communication strategy, and obtain the set communication result. In this way, during the training of the model, no matter how the network fluctuates, the best communication strategy can be determined according to the current bandwidth when the set communication between nodes is performed, so that the bandwidth of the two communication domains can be maximally utilized, and the problem of resource waste does not occur, the efficiency of set communication is improved, and the overall training efficiency of the model is improved.

[0247] It should be noted that the data communication method of the present application is introduced by taking 2 communication domains as an example in the embodiment of FIG. 5, and in specific implementation, more communication domains can also be included, and the communication amount and communication strategy of each communication domain can also be obtained according to the description of S520 and S530, which will not be described one by one here.

[0248] It should be understood that the data communication method of the present application is introduced by taking the reducescatter algorithm as an example in the embodiment of FIG. 5, and for other set communication algorithms, such as allgather and allreduce algorithms, the communication amount and communication strategy of each communication domain can also be determined according to the required set communication result of each node, according to the description of S520 and S530, and according to the system structure information and bandwidth ratio, which will not be described one by one here.

[0249] It should be noted that the data communication method provided in the present application can be applied to collective communication between nodes. Generally, the collective communication occurs in the model training phase, and using the data communication method provided in the present application can improve the training efficiency of the model. In some special cases, the collective communication can also occur in the model inference phase, and using the data communication method provided in the present application can also improve the inference efficiency of the model. Here, examples are not given one by one.

[0250] In summary, the data communication method provided in the present application, when the training system starts to train the model in a distributed manner, if the nodes need to perform collective communication, each node can determine the first communication amount of the node in the first communication domain and the second communication amount of the node in the second communication domain according to the bandwidth ratio between the first communication domain and the second communication domain and the system structure information of the training system, so that the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, and then determines the first communication strategy used by the first communication domain based on the first communication amount, and determines the second communication strategy used by the second communication domain based on the second communication amount, and then completes the collective communication based on the first communication strategy and the second communication strategy, so that each node can maximize the utilization of the bandwidth of the first communication domain and the second communication domain during the collective communication. Even if the bandwidth changes, the node can also adjust the communication strategy in time, so that the communication domain with larger bandwidth can bear more data, and the communication domain with smaller bandwidth can bear less data, avoiding the waste of bandwidth resources and improving the communication efficiency of the collective communication.

[0251] The data communication method and the training system provided in the present application are described in detail above, and the structure of the node provided in the present application is described below in combination with FIG. 13.

[0252] FIG. 13 is a structural schematic diagram of a computing node provided in the present application. The node 1300 can be the first node in the foregoing, and the node 1300 is one node in the training system 200 shown in FIGS. 2-4, such as the node 11 or the first node in the foregoing.

[0253] As shown in FIG. 13, the node 1300 includes an acquisition unit 1310, a calculation unit 1320, a communication unit 1330, and a training unit 1340. It should be understood that FIG. 13 is an exemplary division manner, and the node 1300 can be divided into more or fewer unit modules, which are not specifically limited in the present application.

[0254] The acquisition unit 1310, the calculation unit 1320, the communication unit 1330, and the training unit 1340 can all be implemented by software or by hardware. For example, the implementation of the acquisition unit 1310 will be described in detail below. Similarly, the implementation of the calculation unit 1320, the communication unit 1330, and the training unit 1340 can refer to the implementation of the acquisition unit 1310.

[0255] As an example of a software functional unit, the acquisition unit 1310 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the acquisition unit 1310 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or in different AZs, each AZ including a data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0256] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set up in one region, and cross-region communication between two VPCs in the same region or between VPCs in different regions needs to set up a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0257] As an example of a hardware functional unit, the obtaining unit 1310 can include at least one computing device, such as a server or the like. Alternatively, the obtaining unit 1310 can also be a device implemented by a central processing module (CPU), or a device implemented by an application-specific integrated circuit (ASIC), or a device implemented by a programmable logic device (PLD), and the like. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing module (DPU), a neural network processing module (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0258] The multiple computing devices included in the obtaining unit 1310 can be distributed in the same region or in different regions. The multiple computing devices included in the obtaining unit 1310 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the obtaining unit 1310 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, and the like.

[0259] It should be noted that in other embodiments, the steps implemented by the obtaining unit 1310, the computing unit 1320, the communication unit 1330, and the training unit 1340 can be specified as needed, and the entire function of the training system 200 can be implemented by the obtaining unit 1310, the computing unit 1320, the communication unit 1330, and the training unit 1340 respectively implementing different steps in the data communication method.

[0260] The functions of the obtaining unit 1310, the computing unit 1320, the communication unit 1330, and the training unit 1340 will be explained below.

[0261] The computing node 1300 is a computing node in a distributed training system, the distributed training system including a plurality of communication groups, each communication group including a plurality of computing nodes, the computing nodes in the same communication group implementing data communication through a first communication domain, and the computing nodes in different communication groups implementing data communication through a second communication domain. The distributed training system is specifically described with reference to the embodiments of FIGS. 2-4, which are not repeated here.

[0262] The obtaining unit 1310 is configured to obtain a bandwidth ratio between the first communication domain and the second communication domain, and system structure information of the distributed training system, the system structure information including a number N of communication groups of the distributed training system and a number M of nodes in each communication group, and specifically used to implement S510 and optional steps thereof in the embodiment of FIG. 5.

[0263] The computing unit 1320 is configured to obtain a first communication amount and a second communication amount based on the bandwidth ratio and the system structure information, the first communication amount being a total amount of communication of the first node with other nodes in the communication group through the first communication domain, and the second communication amount being a total amount of communication of the first node with nodes in other communication groups through the second communication domain, a difference between a ratio of the first communication amount and the second communication amount and the bandwidth ratio being less than a threshold, and specifically used to implement S520 and optional steps thereof in the embodiment of FIG. 5.

[0264] The communication unit 1330 is configured to perform collective communication with other nodes in the communication group according to the first communication amount based on the first communication domain, and perform collective communication with nodes in other communication groups according to the second communication amount based on the second communication domain, to obtain a collective communication result, and specifically used to implement S540A-S540B and optional steps thereof in the embodiment of FIG. 5.

[0265] The training unit 1340 is configured to complete model training based on the collective communication result, and specifically used to implement S550 and optional steps thereof in the embodiment of FIG. 5.

[0266] In a possible implementation, the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, including that the ratio of the first communication amount and the second communication amount is the same as the bandwidth ratio, or the ratio of the first communication amount and the second communication amount is approximately the same as the bandwidth ratio, or a difference between the ratio of the first communication amount and the second communication amount and the bandwidth ratio is less than a threshold.

[0267] In a possible implementation, the computing unit 1320 is configured to determine, based on the first communication amount, at least one first communication node and first data to be sent by each first communication node, and determine, based on the second communication amount, at least one second communication node and second data to be sent by each second communication node, the first communication node belonging to the same communication group as the first node, and the second communication node belonging to a different communication group from the first node, and specifically configured to implement S530 and optional steps thereof in the embodiment of FIG. 5; and the communication unit 1330 is configured to obtain, from each first communication node, the first data through the first communication domain, and obtain, from each second communication node, the second data through the second communication domain, to obtain the set communication result.

[0268] In a possible implementation, the data on each node in the plurality of computing nodes is divided into a plurality of sub-data according to the total number of nodes, each sub-data corresponds to a data number, the set communication result includes sub-data with a first number on each node, and the first data and the second data include sub-data with the first number.

[0269] In a possible implementation, the computing unit 1320 is configured to determine, based on the system structure information, at least one first communication node, wherein the first communication node includes all nodes in the communication group in which the first node is located, except the first node; and determine, based on the first communication amount, first data to be sent by each first communication node, wherein the first data includes one or more sub-data, and a difference between a data number of each sub-data and the first number is a multiple of M.

[0270] In a possible implementation, the computing unit 1320 is configured to determine the second data, the second data including sub-data with the first number; determine, based on the second communication amount and the number N of communication groups, a number h1 of full communication nodes and a number h2 of single communication nodes, wherein the second communication node includes the full communication node and the single communication node, the full communication node is in the communication group in which all nodes communicate with the first node, and the single communication node is in the communication group in which only the single communication node communicates with the first node; and determine, based on the number h1 of full communication nodes and the number h2 of single communication nodes, in combination with the system structure information, at least one second communication node.

[0271] In a possible implementation, the computing unit 1320 is configured to determine, as the full communication node, all nodes in h1 / M communication groups after a first communication group in which the first node is located; and determine, as the single communication node, an i th node in h2 communication groups after a communication group in which a last full communication node is located, wherein the first node is an i th node in the first communication group.

[0272] In a possible implementation, the computing unit 1320 is configured to match the bandwidth ratio and the system structure information with the mapping relationship, and the computing unit 1320 is configured to determine the first communication node, the first data, the second communication node, and the second data based on a communication strategy corresponding to the first bandwidth ratio and the system structure information in a case where the mapping relationship includes the system structure information and a gap between the bandwidth ratio and the first bandwidth ratio in the mapping relationship is less than a threshold value, and the computing unit 1320 is configured to obtain the first communication amount and the second communication amount based on the bandwidth ratio and the system structure information in a case where the mapping relationship does not include the system structure information, or the mapping relationship includes the system structure information but does not include the first bandwidth ratio.

[0273] In a possible implementation, the algorithm used by the collective communication includes one or more of a reducescatter algorithm, an allreduce algorithm, and an allgather algorithm.

[0274] In conclusion, the computing node provided in this application can determine the first communication amount of the node in the first communication domain and the second communication amount of the node in the second communication domain according to the bandwidth ratio between the current first communication domain and the second communication domain and the system structure information of the training system when the training system starts to train the model and the nodes need to perform the collective communication, so that the ratio of the first communication amount and the second communication amount is adapted to the bandwidth ratio, and then the first communication strategy used by the first communication domain is determined based on the first communication amount, the second communication strategy used by the second communication domain is determined based on the second communication amount, and then the collective communication is completed based on the first communication strategy and the second communication strategy, so that each node can maximize the utilization of the bandwidth of the first communication domain and the second communication domain in the collective communication process, and even if the bandwidth changes, the node can also adjust the communication strategy in time, so that the communication domain with large bandwidth can undertake more data and the communication domain with small bandwidth can undertake less data, thereby avoiding the waste of bandwidth resources and improving the communication efficiency of the collective communication.

[0275] The data communication method, the training system, and the computing node in the system provided in this application are described in detail above, and the computing device provided in this application is explained and described below with reference to FIG. 14 and FIG. 15.

[0276] FIG. 14 is a structural schematic diagram of a computing device provided by the present application. As shown in FIG. 14, the computing device 1400 includes a bus 1402, a processor 1404, a memory 1406, an acceleration card 1407, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate through the bus 1402. The computing device 1400 can be a server. It should be understood that the number of processors, memories, and acceleration cards in the computing device 1400 is not limited by the present application. The computing device 1400 can be used to deploy the computing nodes in the foregoing embodiments of FIGS. 1-13. The memory 1406 is configured to store instructions, and the processor 1404 is configured to execute the instructions stored in the memory 1406. The memory 1406 stores program codes, and the processor 1404 can invoke the program codes stored in the memory 1406 to perform the following operations:

[0277] obtaining a bandwidth ratio between the first communication domain and the second communication domain, and system structure information of the distributed training system, the system structure information including a number N of communication groups of the distributed training system and a number M of nodes in each communication group;

[0278] obtaining the first communication amount and the second communication amount based on the bandwidth ratio and the system structure information, wherein the first communication amount is a total communication amount of the first node communicating with other nodes in the communication group through the first communication domain, the second communication amount is a total communication amount of the first node communicating with nodes in other communication groups through the second communication domain, and a difference between a ratio of the first communication amount and the second communication amount and the bandwidth ratio is less than a threshold value;

[0279] obtaining a collection communication result based on the first communication domain, performing collection communication with other nodes in the communication group according to the first communication amount, and based on the second communication domain, performing collection communication with nodes in other communication groups according to the second communication amount;

[0280] completing model training based on the collection communication result.

[0281] It should be understood that the memory 1406 stores executable program codes, and the processor 1404 executes the executable program codes to implement the operation steps of the data communication method described in each of the foregoing embodiments of FIGS. 5-12. The program codes can include one or more software units, and the one or more software units are the obtaining unit, the calculating unit, the communicating unit, and the training unit in the embodiment of FIG. 13, wherein the obtaining unit is configured to implement S510 and optional steps thereof in the embodiment of FIG. 5, the calculating unit is configured to implement S520, S530, and optional steps thereof in the embodiment of FIG. 5, the communicating unit is configured to implement S540A, S540B, and optional steps thereof in the embodiment of FIG. 5, and the training unit is configured to implement S550 and optional steps thereof in the embodiment of FIG. 5. Details of each unit are not described herein.

[0282] The bus 1402 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (UBus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The unified bus can also be referred to as a coherent accelerated processor bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is represented in FIG. 14, but it does not mean that there is only one bus or only one type of bus. The bus 1402 can include a path for transmitting information between various components (e.g., the memory 1406, the processor 1404, the communication interface 1408) of the computing device 1400. The unified bus can also be referred to as a coherent accelerated processor bus.

[0283] The processor 1404 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, a FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, or the like computing device.

[0284] The memory 1406 can include a volatile memory, such as a random access memory (RAM) including a DRAM and / or SRAM. The memory 1406 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). In addition, the memory 1406 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0285] It is worth mentioning that the same type of storage medium can be configured in the same computing device to implement the function of the memory 1406, or two or more types of storage media can be configured to implement the function of the memory 1406, which is not limited in the present application.

[0286] The communication interface 1408 uses a transceiver module such as but not limited to a network interface card, a transceiver, etc. to realize the communication between the computing device 1400 and other devices or communication networks.

[0287] As a possible implementation manner, the method shown in FIGS. 5-12 can be implemented by the acceleration card 1407 in the computing device 1400, in addition to being implemented by the processor in the computing device 1400 executing the program code in the memory, wherein the acceleration card 1407 can be implemented by a computing device such as a DPU, NPU, GPU, XPU, SoC, etc. or a computing unit such as an AI chip, which is not limited in the present application. The acceleration card 1407 includes a processing unit and a power supply circuit, and the power supply circuit is used to power the processing unit so that the processing unit executes the operation steps of the data communication method described in each embodiment of FIGS. 5-12.

[0288] As a possible implementation manner, in addition to being implemented by the processor in the computing device 1400 executing the program code in the memory, or by the acceleration card 1407, the method described in FIGS. 5-12 can also be implemented by the combination of the processor 1404 and the acceleration card 1407 in the computing device 1400 to execute the operation steps performed by the corresponding subject in the data communication method described in FIGS. 5-12. Specifically, the program code stored in the memory 1406 includes part of the unit modules shown in FIG. 13, so that the processor 1404 executes the program code to implement part of the operation steps of the data communication method, and the power supply circuit in the acceleration card 1407 powers the processing unit so that the processing unit executes another part of the operation steps.

[0289] For example, the memory 1406 stores the program code of the obtaining unit, and the processing unit of the acceleration card 1407 is used to implement the functions of the computing unit, the communication unit and the training unit. The above examples are used for illustration, and the application is not limited thereto, and the function descriptions of the above-mentioned units can refer to the related content described above, which will not be repeated and described here.

[0290] The embodiments of the present application also provide a computing device cluster, as shown in FIG. 15, which is an example diagram of a computing device cluster provided by the present application, which can include at least one computing device of the embodiment of FIG. 14, and the computing device cluster is used to deploy the training system 200 described above.

[0291] As a possible implementation, the memory 1406 in one or more of the computing devices 1400 in the computing device cluster can have stored therein instructions which are the same as instructions to implement operational steps of the data communication method described in FIG. 5 to FIG. 12. Alternatively, the memory 1406 in one or more of the computing devices 1400 can have stored therein instructions which are different from each other and which are respectively to implement operational steps of the data communication method described in FIG. 5 to FIG. 12, in other words, a combination of one or more of the computing devices 1400 can collectively execute instructions to implement operational steps required for implementing the data communication method.

[0292] As a possible implementation, the accelerator card 1407 in one or more of the computing devices 1400 in the computing device cluster can have stored therein instructions which are the same as instructions to implement operational steps of the data communication method described in FIG. 5 to FIG. 12. Alternatively, the accelerator card 1407 in one or more of the computing devices 1400 can have stored therein instructions which are different from each other and which are respectively to implement operational steps of the data communication method described in FIG. 5 to FIG. 12, in other words, a combination of one or more of the computing devices 1400 can collectively execute instructions to implement operational steps required for implementing the data communication method.

[0293] As a possible implementation, the memory 1406 in one or more of the computing devices 1400 in the computing device cluster can have stored therein instructions which are the same as instructions to implement a part of operational steps of the data communication method described in FIG. 5 to FIG. 12, and the accelerator card 1407 can have stored therein instructions which are the same as instructions to implement another part of operational steps. Alternatively, the memory 1406 in one or more of the computing devices 1400 can have stored therein instructions which are different from each other and which are respectively to implement a part of operational steps of the data communication method described in FIG. 5 to FIG. 12, and the accelerator card 1407 can have stored therein instructions which are different from each other and which are respectively to implement another part of operational steps, in other words, a combination of one or more of the computing devices 1400 can collectively execute instructions to implement operational steps required for implementing the data communication method.

[0294] The embodiment of the present application further provides a chip, which comprises a processing unit and a power supply circuit, and the power supply circuit is used for powering the processing unit, so that the processing unit executes operational steps of a first node in the data communication method in the embodiments of FIG. 5 to FIG. 12, and details are not described herein for the sake of brevity.

[0295] The embodiment of the present application further provides a chip system, which comprises a plurality of chips described above, and the chip system is used for deploying the training system in the foregoing description.

[0296] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the data communication method provided by the present application.

[0297] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the inference method based on the large language model or instruct the computing device to perform the inference method.

[0298] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data communication, characterized by, The method is applied to a distributed training system, the distributed training system comprising a plurality of communication groups, each communication group comprising a plurality of computing nodes, computing nodes in a same communication group realizing data communication through a first communication domain, and computing nodes in different communication groups realizing data communication through a second communication domain, the method comprising: a first node in the plurality of computing nodes acquiring a bandwidth ratio between the first communication domain and the second communication domain, and system structure information of the distributed training system; the first node acquiring a first communication volume and a second communication volume based on the bandwidth ratio and the system structure information, wherein the first communication volume is a total communication volume of the first node communicating with other nodes in a communication group through the first communication domain, and the second communication volume is a total communication volume of the first node communicating with nodes in other communication groups through the second communication domain, wherein a communication volume ratio between the first communication volume and the second communication volume is adapted to the bandwidth ratio; the first node performing collective communication with other nodes in the communication group according to the first communication volume based on the first communication domain, and performing collective communication with nodes in other communication groups according to the second communication volume based on the second communication domain, to obtain a collective communication result; the first node performing distributed training of an artificial intelligence (AI) model based on the collective communication result.

2. The method of claim 1, wherein, The communication volume ratio between the first communication volume and the second communication volume is adapted to the bandwidth ratio, including that the communication volume ratio is the same as the bandwidth ratio, or the communication volume ratio is approximately the same as the bandwidth ratio, or a difference between the communication volume ratio and the bandwidth ratio is less than a threshold value.

3. The method of claim 2, wherein, The first node performing collective communication with other nodes in the communication group according to the first communication volume based on the first communication domain, and performing collective communication with nodes in other communication groups according to the second communication volume based on the second communication domain, to obtain a collective communication result, includes: the first node determining at least one first communication node and first data to be sent by each first communication node based on the first communication volume, and determining at least one second communication node and second data to be sent by each second communication node based on the second communication volume, the first communication node belonging to a same communication group as the first node, and the second communication node belonging to a different communication group from the first node; the first node acquiring the first data from each first communication node through the first communication domain, and acquiring the second data from each second communication node through the second communication domain, to obtain the collective communication result.

4. The method of claim 3, wherein, Data on each node in the plurality of computing nodes is divided into a plurality of sub-data according to a total number of nodes, each sub-data corresponding to a data number, the collective communication result including sub-data with a first number as a data number on each node, and the first data and the second data including sub-data with the first number as the data number.

5. The method of claim 4, wherein, The first node determining at least one first communication node and first data to be sent by each first communication node based on the first communication volume includes: The first node determines at least one first communication node based on the system structure information, wherein the first communication node comprises all nodes in a communication group except the first node; The first node determines first data to be sent by each first communication node based on the first traffic, wherein the first data comprises one or more sub-data, and a difference between a data number of each sub-data and the first number is a multiple of M, wherein M is a number of nodes in each communication group.

6. The method according to claim 4 or 5, characterized in that, The first node determines at least one second communication node based on the second traffic, and second data to be sent by each second communication node comprises: The first node determines second data, wherein the second data comprises sub-data with the first number; The first node determines a number h1 of full communication nodes and a number h2 of single communication nodes based on the second traffic and a number N of the communication groups, wherein the second communication node comprises the full communication node and the single communication node, all nodes in a communication group where the full communication node is located communicate with the first node, and only the single communication node in a communication group where the single communication node is located communicates with the first node; The first node determines the at least one second communication node based on the number h1 of the full communication nodes and the number h2 of the single communication nodes in combination with the system structure information.

7. The method of claim 6, wherein, The first node determines the at least one second communication node based on the number h1 of the full communication nodes and the number h2 of the single communication nodes in combination with the system structure information comprises: The first node determines all nodes in h1 / M communication groups after a first communication group where the first node is located as the full communication node; The first node determines an i-th node in h2 communication groups after a communication group where a last full communication node is located as the single communication node, wherein the first node is an i-th node in the first communication group.

8. The method according to any one of claims 3 to 7, characterized in that, The first node obtains the first traffic and the second traffic based on the bandwidth ratio and the system structure information comprises: The first node matches the bandwidth ratio and the system structure information with a mapping relationship; The first node determines the first communication node, the first data, the second communication node and the second data based on a communication strategy corresponding to the first bandwidth ratio and the system structure information in a case that the mapping relationship comprises the system structure information, and a difference between the bandwidth ratio and the first bandwidth ratio in the mapping relationship is less than a threshold value; The first node obtains the first traffic and the second traffic based on the bandwidth ratio and the system structure information in a case that the mapping relationship does not comprise the system structure information, or the mapping relationship comprises the system structure information but does not comprise the first bandwidth ratio.

9. The method according to any one of claims 1 to 8, characterized in that, The collective communication method comprises one or more of a reducescatter, an allreduce, an allgather, and an all-to-all algorithm.

10. A computing node, characterized in that, The computing node is a computing node in a distributed training system, the distributed training system comprising a plurality of communication groups, each communication group comprising a plurality of computing nodes, computing nodes in a same communication group performing data communication through a first communication domain, and computing nodes in different communication groups performing data communication through a second communication domain, the computing node comprising: an obtaining unit, configured to obtain a bandwidth ratio between the first communication domain and the second communication domain, and system structure information of the distributed training system; a computing unit, configured to obtain a first communication amount and a second communication amount based on the bandwidth ratio and the system structure information, the first communication amount being a total communication amount of the first node performing communication with other nodes in a communication group through the first communication domain, the second communication amount being a total communication amount of the first node performing communication with nodes in other communication groups through the second communication domain, a ratio of the first communication amount to the second communication amount being adapted to the bandwidth ratio; a communication unit, configured to perform collective communication with other nodes in the communication group based on the first communication amount through the first communication domain, and perform collective communication with nodes in other communication groups based on the second communication amount through the second communication domain, to obtain a collective communication result; a training unit, configured to perform distributed training of an artificial intelligence (AI) model based on the collective communication result.

11. The computing node of claim 10, wherein, The ratio of the first communication amount to the second communication amount is adapted to the bandwidth ratio, which comprises that the ratio of the first communication amount to the second communication amount is the same as the bandwidth ratio, or the ratio of the first communication amount to the second communication amount is approximately the same as the bandwidth ratio, or a difference between the ratio of the first communication amount to the second communication amount and the bandwidth ratio is less than a threshold value.

12. The computing node of claim 11, wherein the computing unit is configured to determine at least one first communication node and first data to be sent by each first communication node based on the first communication amount, and determine at least one second communication node and second data to be sent by each second communication node based on the second communication amount, the first communication node belonging to a same communication group as the first node, and the second communication node belonging to a different communication group from the first node; the communication unit is configured to obtain the first data from each first communication node through the first communication domain, and obtain the second data from each second communication node through the second communication domain, to obtain the collective communication result.

13. The computing node of claim 12, wherein, The data on each node of the plurality of computing nodes is divided into a plurality of sub-data according to a total number of nodes, each sub-data corresponding to a data number, the collective communication result comprises sub-data with a first number on each node, and the first data and the second data comprise sub-data with the first number.

14. The computing node of claim 13, wherein determine at least one first communication node based on the system structure information, wherein the first communication node comprises all nodes in a communication group in which the first node is located except the first node; determine first data to be sent by each first communication node based on the first traffic, wherein the first data comprises one or more sub-data, and a difference between a data number of each sub-data and the first number is a multiple of M, where M is a number of nodes in each communication group.

15. The computing node of claim 13 or 14, wherein determine second data, wherein the second data comprises sub-data with a data number of the first number; determine a number h1 of full communication nodes and a number h2 of single communication nodes based on the second traffic and a number N of the communication groups, wherein the second communication node comprises the full communication nodes and the single communication nodes, all nodes in a communication group in which the full communication nodes are located communicate with the first node, and only the single communication nodes in a communication group in which the single communication nodes are located communicate with the first node; determine the at least one second communication node based on the number h1 of the full communication nodes and the number h2 of the single communication nodes in combination with the system structure information.

16. The computing node of claim 15, wherein determine that all nodes in h1 / M communication groups after a first communication group in which the first node is located are the full communication nodes; determine that an i-th node in h2 communication groups after a communication group in which a last full communication node is located is a single communication node, wherein the first node is the i-th node in the first communication group.

17. The computing node of any one of claims 12 to 16, wherein match the bandwidth ratio and the system structure information with a mapping relationship; determine the first communication node, the first data, the second communication node, and the second data based on a communication strategy corresponding to the first bandwidth ratio and the system structure information in a case that the mapping relationship comprises the system structure information, and a difference between the bandwidth ratio and a first bandwidth ratio in the mapping relationship is less than a threshold value; obtain the first traffic and the second traffic based on the bandwidth ratio and the system structure information in a case that the mapping relationship does not comprise the system structure information, or the mapping relationship comprises the system structure information but does not comprise the first bandwidth ratio.

18. The compute node of any of claims 10 to 17, wherein, The algorithm used by the collective communication comprises one or more of a reducescatter algorithm, an allreduce algorithm, an allgather algorithm, and an all-to-all algorithm.

19. A computing device, comprising: The computing device comprises a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operational steps of the method of any of claims 1-8.

20. A distributed training system, comprising: The distributed training system comprises a plurality of communication groups, each communication group comprising a plurality of computing devices, computing devices within a same communication group communicate data via a first communication domain, computing devices within different communication groups communicate data via a second communication domain, each computing device comprises a processor and a memory, the processor of each computing device is configured to execute instructions stored in the memory to cause each computing device to perform the operational steps of the method of any of claims 1-9.

21. A chip, characterized by The chip comprises a power supply unit and a processing unit, the power supply unit is configured to supply power to the processing unit, so that the processing unit implements the operational steps of the method of any of claims 1-9.

22. A computer-readable storage medium, characterized in that, Computer program instructions are included, when the computer program instructions are executed by a computing device, the computing device performs the operational steps of the method of any of claims 1-9.

23. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computing device, cause the computing device to perform the operational steps of the method of any of claims 1-9.

Images