Communication ring generation method, model training method, network system, device, medium, and program product

Abstract

In the present disclosure, the number of various types of communication rings to be generated is determined on the basis of ring type information of various types of communication rings to be created within each working node in a network, working node information of the network, a set communication ring bandwidth, and a set parameter representing the number of the various types of communication rings to be generated; on the basis of the number of processors corresponding to and required for creating the various types of communication rings and the number of the communication rings to be generated within each working node, for each of multiple working nodes, an adaptive number of processors are selected from among multiple processors within the working node and are connected into a ring, so that multiple node-level communication rings are respectively generated within the multiple working nodes; and then the multiple node-level communication rings within the multiple working nodes are connected to construct multiple cross-node communication rings.

Description

Communication ring generation method, model training method, network system, device, medium, and program product Cross-reference to related applications

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411099431.8, filed on August 9, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the field of computer technology, and particularly relates to a communication ring generation method, a model training method, a network system, an electronic device, a storage medium, and a program product. BACKGROUND

[0003] With the increasing model size of artificial intelligence models, the number of model parameters is also increasing. In order to better accelerate the model training speed under this trend, a distributed training method is often used to train the model. Distributed training processes data in parallel on multiple computing nodes, enabling the model to converge faster. In the distributed training network system architecture, a collective communication algorithm is used to achieve effective communication between multiple computing nodes, such as the Ring algorithm, which is a parallel scheduling algorithm based on ring structure. The Ring algorithm usually assumes that the bandwidth within the computing node is sufficient and the network bandwidth between the computing nodes is limited. It first performs ring communication on GPUs within each computing node, and then performs ring communication on all computing nodes. Specifically, it directly connects the communication rings formed by the GPUs of different computing nodes to form a larger communication ring. When the bandwidth between the computing nodes is more sufficient, the above existing Ring algorithm has the problem that the bandwidth between the computing nodes cannot be fully utilized, and the bandwidth within the computing node becomes a bottleneck. SUMMARY

[0004] The present disclosure provides a communication ring generation method, a model training method, a network system, an electronic device, a storage medium, and a program product that solve the above problems or at least partially solve the above problems.

[0005] In a first embodiment, the disclosure provides a communication ring generation method. The method comprises: obtaining ring type information of multiple types of communication rings that need to be created in each working node in a network; wherein the ring type information of a type of communication ring can reflect the number of processors that need to be used when creating the type of communication ring, and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information including a first value representing the occupation of bandwidth in a node and a second value representing the occupation of network bandwidth between nodes; the network includes multiple working nodes; determining the number of various types of communication rings that need to be generated in each working node according to the bandwidth occupation information of various types of communication rings, the working node information of the network, and the set communication ring bandwidth, and a parameter representing the number of various types of communication rings that need to be generated in each working node; based on the number of processors that need to be used when creating the various types of communication rings and the number of various types of communication rings that need to be generated in each working node, selecting an adaptive number of processors from multiple processors in each working node for ring connection operation to generate multiple node-level communication rings in each working node; connecting multiple node-level communication rings between the multiple working nodes to construct multiple cross-node communication rings of the network; wherein the total number of the multiple cross-node communication rings is equal to the total number of the multiple node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring of the same type from the multiple working nodes and sequentially connecting the selected node-level communication rings.

[0006] In a second embodiment, the disclosure also provides a communication ring generation method. The method comprises: obtaining working node information of a network; wherein the network comprises a plurality of working nodes, and a plurality of node-level communication rings are generated in the plurality of working nodes; determining a communication ring connection strategy between the plurality of working nodes according to the working node information; connecting the plurality of node-level communication rings between the plurality of working nodes based on the communication ring connection strategy; and constructing a plurality of cross-node communication rings of the network. One node-level communication ring is formed by connecting at least part of processors in a plurality of processors in a corresponding working node in a ring shape. The plurality of node-level communication rings comprise a plurality of types of communication rings, and the number of each type of communication ring is determined according to bandwidth occupation information corresponding to each type of communication ring, working node information of the network, and a set communication ring bandwidth, and a parameter representing the number of each type of communication ring to be generated in each working node. The bandwidth occupation information corresponding to one type of communication ring includes the bandwidth occupation in the node and the network bandwidth occupation between nodes when transmitting data through the type of communication ring. The total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings. One cross-node communication ring is formed by selecting one node-level communication ring belonging to the same type of communication ring from the plurality of working nodes and sequentially connecting the selected node-level communication rings in a head-to-tail manner. The working node information comprises the total number of working nodes and the node number of each working node.

[0007] In a third embodiment, the disclosure provides a model training method. The model training method comprises: obtaining a training sample data set; dividing the training sample data set into a total number of data blocks according to the total number of a plurality of cross-node communication rings in a network; assigning each cross-node communication ring in the plurality of cross-node communication rings with one data block, so that each cross-node communication ring trains a target model using the assigned data block; and optimizing the target model according to the training result of each cross-node communication ring on the target model. The network comprises a plurality of working nodes, and a plurality of node-level communication rings are generated in each working node and the target model is deployed in each working node. The plurality of node-level communication rings comprise different types of communication rings. One node-level communication ring is formed by connecting at least part of processors in a plurality of processors in a corresponding working node in a ring shape. One cross-node communication ring is formed by selecting one node-level communication ring belonging to the same type of communication ring from the plurality of working nodes and sequentially connecting the selected node-level communication rings in a head-to-tail manner. The total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings.

[0008] In a fourth embodiment, the present disclosure provides a network system. The system comprises: a plurality of worker nodes; the worker nodes comprise a plurality of processors; a control node configured to obtain ring type information of a plurality of types of communication rings to be created in each worker node; wherein the ring type information of a type of communication ring is capable of reflecting a number of processors to be used when creating the type of communication ring, and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information comprising a first value representing occupied bandwidth in a node and a second value representing occupied network bandwidth between nodes; determining a number of the plurality of types of communication rings to be generated in each worker node according to the bandwidth occupation information of the plurality of types of communication rings, worker node information of the network, and a set communication ring bandwidth, and a parameter representing the number of the plurality of types of communication rings to be generated in each worker node; performing, for the plurality of worker nodes, a ring connection operation of selecting an adaptive number of processors from a plurality of processors in a worker node, to generate a plurality of node-level communication rings in the plurality of worker nodes, respectively, based on the number of processors to be used when creating the plurality of types of communication rings and the number of the plurality of types of communication rings to be generated in each worker node; and connecting the plurality of node-level communication rings between the plurality of worker nodes to construct a plurality of cross-node communication rings of the network; wherein a total number of the plurality of cross-node communication rings is equal to a total number of the plurality of node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring of the same type from the plurality of worker nodes and sequentially connecting the selected node-level communication rings.

[0009] In a fifth embodiment, the present disclosure provides an electronic device. The electronic device comprises a memory and a processor, wherein the memory is configured to store a program; and the processor is coupled to the memory and configured to execute the program stored in the memory to implement steps in each method embodiment provided by the present disclosure.

[0010] In a sixth embodiment, the present disclosure provides a computer-readable storage medium. The computer-readable storage medium stores a computer program; when the computer program is executed by a processor, steps in each method embodiment provided by the present disclosure can be implemented.

[0011] In a seventh embodiment, the present disclosure provides a computer program product. The computer program product comprises a computer program, when the computer program is executed by a processor, the processor is capable of implementing steps in each method embodiment provided by the present disclosure.

[0012] In the technical solutions provided in the embodiments of the present disclosure, when establishing a communication ring in each working node in a network, the ring type information of multiple types of communication rings that need to be created in each working node in the network is first acquired; the ring type information of one type of communication ring can reflect the number of processors that need to be used when creating the type of communication ring and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information including a first value representing the occupied bandwidth in a node and a second value representing the occupied network bandwidth between nodes; the network includes multiple working nodes; then, the number of various types of communication rings that need to be generated in each working node is determined according to the ring type information of various types of communication rings, the working node information of the network, and the set communication ring bandwidth and the parameter representing the number of various types of communication rings that need to be generated in each working node, which can effectively ensure that the ratio of the in-node bandwidth used by the communication ring to the inter-node network bandwidth is close to the bandwidth ratio of the actual hardware in the network (i.e., the actual ratio of the inter-working node bandwidth to the in-working node bandwidth), thereby improving the bandwidth utilization. Further, based on the number of processors that need to be used when creating the various types of communication rings and the number of various types of communication rings that need to be generated in each working node, the operation of selecting an adaptive number of processors from multiple processors in each working node for ring connection is performed for the multiple working nodes to generate multiple node-level communication rings in the multiple working nodes, respectively. The multiple node-level communication rings generated in the working nodes in the present solution are of multiple types, and the available bandwidth can be effectively utilized through the multiple types of communication rings. Further, the multiple node-level communication rings between the multiple working nodes are connected to construct multiple cross-node communication rings of the network. In a specific example, when constructing the cross-node communication rings of the network, the communication ring connection strategy of the multiple working nodes is determined according to the working node information of the network (including the total number of working nodes and the node number of each working node), and then the multiple node-level communication rings between the multiple working nodes are connected based on the communication ring connection strategy to construct the multiple cross-node communication rings of the network. The total number of the multiple cross-node communication rings is equal to the total number of the multiple node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring of the same type from the multiple working nodes and sequentially connecting the selected node-level communication rings, which enables full utilization of the inter-node bandwidth and improves the collective communication capability when the multiple cross-node communication rings are used to process data (such as model training) in parallel, thereby shortening the data processing time. In addition, in the cross-node communication ring, the two processors connected between at least part of the two working nodes in the multiple working nodes can also have the same number, which enables the communication between at least part of the two nodes to occur only between the processors with the same number, thereby effectively avoiding inefficient cross-working node communication and improving the communication performance.For example, the processors with the same number in different worker nodes have basically the same deployment position in the worker nodes, and the performance is also basically the same affected by other devices around, so the performance difference is usually small, so that the communication between the two nodes occurs between the processors with the same number, which is conducive to effectively avoiding inefficient cross-worker node communication.

[0013] The model training can be performed using the plurality of cross-node level communication rings constructed above. Specifically, the training sample data set can be divided into the total number of data blocks according to the total number of the plurality of cross-node communication rings, so as to allocate a data block to each cross-node communication ring, and each cross-node communication ring trains the target model using the allocated data block, so as to optimize the target model according to the training results of the target model by each cross-node communication ring. It can be seen that the parallel processing of the training data is realized when the model is trained by the plurality of cross-node communication rings, which can effectively ensure that the inter-machine bandwidth is fully utilized and the collective communication capability is improved, so as to achieve the effect of expanding the training capability and shortening the training time. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0015] Fig. 1 is a structural schematic diagram of a network system architecture for distributed training provided by an exemplary embodiment of the present disclosure.

[0016] Fig. 2 is a schematic diagram of a link topology structure between GPUs in a computing node provided by an exemplary embodiment of the present disclosure.

[0017] Fig. 3a shows a cross-node communication ring constructed using a 7:1 type communication ring in the case that the total number of worker nodes in the network is even.

[0018] Fig. 3b shows a cross-node communication ring constructed using a 7:1 type communication ring in the case that the total number of worker nodes in the network is odd.

[0019] Fig. 4a shows a cross-node communication ring constructed using a 6:2 type communication ring in the case that the total number of worker nodes in the network is even.

[0020] Fig. 4b shows a cross-node communication ring constructed using a 6:2 type communication ring in the case that the total number of worker nodes in the network is odd.

[0021] Figure 5a shows a cross-node communication ring constructed using a 4:4 type communication ring when the total number of worker nodes in the network is even.

[0022] Figure 5b shows a cross-node communication ring constructed using a 4:4 type communication ring when the total number of worker nodes in the network is odd.

[0023] Figures 6 and 7 are flow diagrams of the communication ring generation method shown in the present disclosure.

[0024] Figures 8 and 9 are structural diagrams of the communication ring generation apparatus shown in the present disclosure.

[0025] Figure 10 is a structural diagram of an electronic device provided in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] With the development of artificial intelligence technology, artificial intelligence models (such as deep learning models) are becoming more and more complex, and the number of model parameters of the artificial intelligence models is increasing. Under this trend, model training usually adopts distributed model training to improve the training speed. Distributed model training can effectively expand the training capability and shorten the training time by parallel processing data on multiple computing nodes. Since distributed training involves synchronization of a large amount of data between computing nodes, network communication often becomes a key bottleneck.

[0027] In a distributed training network, network communication often uses collective communication. Collective communication refers to a communication process in which multiple computing nodes in a distributed training computer cluster system complete data exchange and synchronization through a cooperative manner. Common collective communication operations include Broadcast, Gather, AllReduce, and the like. Among them, the Broadcast operation is to broadcast the data on a certain computing node to all other computing nodes. Gather is to collect data to a specified root node, that is, when performing the Gather operation, each computing node sends its data to the specified root node, and the root node is responsible for receiving and storing these data. AllReduce is a global reduction operation, which is to perform aggregation operations (such as summation, maximum value, etc.) on the data (such as gradient values) in the same position on all computing nodes and broadcast the results back to each computing node. In this way, each computing node can obtain the globally aggregated data, so as to update the local model parameters using the globally aggregated data information.

[0028] The current mainstream collective communication library (such as NCCL) often provides the aforementioned collective communication operations such as Broadcast, Gather, AllReduce, etc. NCCL is designed specifically for multi-GPU and multi-computing node environments in deep learning and high-performance computing (HPC) to optimize and accelerate collective communication. NCCL can provide high-performance standard collective communication functions in multi-GPU and multi-computing node architectures, which takes advantage of the high bandwidth and low latency characteristics of GPUs to achieve fast data transmission between multiple GPUs. NCCL supports multiple collective communication algorithms, such as ring-based communication algorithms, halving and doubling communication algorithms, etc. In the ring-based communication algorithm (Ring algorithm), the GPUs participating in the calculation are logically formed into a ring-shaped communication ring; the data is divided into multiple blocks, and then transmitted along the communication ring from one GPU to another; after each GPU receives the data block, it performs necessary operations such as summation, maximum value, minimum value, etc., and passes the result to the next GPU; the above process continues until all data blocks are processed, and finally each GPU obtains the final result. The Ring algorithm used by the current NCCL when grouping the GPUs participating in the calculation into a ring is based on the assumption that the bandwidth within the computing node is sufficient and the network bandwidth between the computing nodes is limited. Under this assumption, first, all GPUs within each computing node are grouped into a ring communication, and then all computing nodes are grouped into a ring communication. Specifically, the communication rings formed by the GPUs of different computing nodes are directly connected end-to-end to form a larger communication ring (a heterogeneous communication ring across computing nodes).For example, in a network system architecture for distributed training, there are a computing node node0 and a computing node node1, and the number of GPUs in each computing node is 8, and using the existing Ring algorithm, first, 8 GPUs in each computing node are grouped into a ring structure according to a certain logical order to obtain a corresponding communication ring, for example, one communication ring Ring0 obtained by grouping the 8 GPUs in the computing node node0 into a ring is: GPU0 (head, the starting GPU of the communication ring) → GPU1 → GPU3 → GPU2 → GPU7 → GPU4 → GPU6 → GPU5 (tail, the terminating GPU of the communication ring), and one communication ring Ring1 obtained by grouping the 8 GPUs in the computing node node1 into a ring is: GPU2 (head) → GPU3 → GPU0 → GPU1 → GPU4 → GPU5 → GPU6 → GPU7 (tail); then, the head and tail of the communication ring Ring0 and the communication ring Ring1 are directly connected (i.e., GPU5 in the communication ring Ring0 is connected with GPU2 in the communication ring Ring1) to establish inter-computing node communication, so as to realize ring grouping communication of the computing node node0 and the computing node node1, and thus form a final communication ring. As can be seen from the example, through the above ring grouping communication, the ratio of inter-computing node bandwidth to intra-computing node bandwidth is 1:7, and when the inter-computing node bandwidth is more sufficient, the existing Ring algorithm cannot fully utilize the inter-computing node bandwidth, and the intra-computing node bandwidth becomes a communication performance bottleneck; and the above method of directly connecting the head and tail of the inter-computing node communication ring will have poor communication performance. For example, if the head and tail are connected to cause the inter-computing node communication to occur between two GPUs with a large performance difference, the inter-computing node communication rate can be low.

[0029] In order to solve the problems existing in the existing Ring algorithm, the present disclosure provides a communication ring generation technical solution, which is realized based on a newly designed set communication algorithm of the present disclosure. The newly designed communication algorithm is a newly designed heterogeneous Ring algorithm, which can adapt to different actual inter-computing node bandwidth and intra-computing node bandwidth ratios, to automatically generate a heterogeneous communication ring Ring structure with better performance according to the actual inter-computing node bandwidth and intra-computing node bandwidth ratios, to effectively utilize the available bandwidth, and to ensure that the communication performance is improved.

[0030] In order for those skilled in the art to better understand the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure.

[0031] In some of the flowcharts described in the specification, claims, and drawings of the present disclosure, a plurality of operations appearing in a particular order include operations that can be performed in the order they appear in this document or in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these flowcharts can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. Also, the term "or / and" in the present disclosure is only a description of the association between the associated objects, which means that there can be three relationships, for example: A or / and B, which means that A can exist alone, A and B exist together, and B alone. The character " / " in the present disclosure generally represents an "or" relationship between the associated objects. It should be noted that the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the goods or systems including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such goods or systems. Without more limitations, the element defined by the statement "including a" does not exclude the existence of another identical element in the goods or systems including the element. In addition, the following embodiments are only some of the embodiments of the present disclosure, not all embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0032] The following describes the technical solutions of the embodiments of the present disclosure.

[0033] It should be noted that the data involved in the present disclosure, such as the training sample data set, is fully authorized data provided by the corresponding data provider, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0034] Before introducing the technical solutions of the embodiments of the present disclosure, a new set communication algorithm based on which the embodiments of the present disclosure are implemented is introduced. The new heterogeneous Ring algorithm is a new heterogeneous Ring algorithm proposed by the scheme design of the present disclosure. Compared with the existing Ring algorithm, the new heterogeneous Ring algorithm can automatically combine the actual bandwidth between the computing nodes in the network and the bandwidth ratio within the computing nodes to generate a heterogeneous communication ring Ring with better network system performance, thereby improving the set communication performance.

[0035] The network system can be, but is not limited to, used for distributed training. For example, FIG. 1 shows a schematic diagram of a network system architecture for distributed training. As shown in FIG. 1, the network system architecture 1 for distributed training includes a plurality of computing nodes, such as N computing nodes, where N is a positive integer greater than 2, and specifically, three computing nodes, including a computing node node0, a computing node node1, and a computing node node2. The computing nodes can be used for computing and processing data during distributed training, and each computing node can include a plurality of GPUs, such as 4 or 8 GPUs. Generally, each computing node includes 8 GPUs. The plurality of GPUs in each computing node are connected using a private protocol, which includes but is not limited to NVLink. NVLink is a bus communication connection protocol using a point-to-point structure and serial transmission, which is mainly used for connecting a central processing unit (CPU) and a graphics processing unit (GPU), and can also be used for connecting a plurality of GPUs. The computing nodes can communicate with each other through a communication network. The communication network can be a wireless or wired network or a combination thereof, such as an Ethernet or Infiniband.

[0036] In implementation, the computing node can be any computing device with computing / processing and communication capabilities, including but not limited to a server. The server can be a standalone physical server, a server cluster formed by a plurality of servers, a cloud server providing cloud services, cloud databases, cloud storage, cloud communication, etc.

[0037] Further, in implementation, the network system can further include a client device. A user can input corresponding information through the client device. For example, the user can input training sample data through the client device, so that the network system performs training on a specific model based on the training sample data; or, the user can input Ring type information that needs to be created in the computing node, such as inputting 7:1, which means that a 7:1 type of communication ring needs to be created in the computing node. For details of the 7:1 type of communication ring, please refer to the relevant content below.

[0038] Specifically, the new heterogeneous Ring algorithm proposed by the disclosed scheme automatically combines to generate a heterogeneous communication ring with better network performance, and the principle is as follows.

[0039] 1. Problem formalization.

[0040] Assuming each computing node in the network system contains 8 GPUs (such as GPU0, GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7), and the node-in total bandwidth (such as NVLink bandwidth) of a computing node is e, and the node-to-node total network bandwidth between two computing nodes is d, then: 11) determine the target is: by forming a ring for each of the 8 GPUs contained in each computing node, so as to generate multiple communication rings in each computing node and multiple communication rings containing different types of communication rings.

[0041] For example, the purpose is to generate a 7:1 type communication ring Ring in each computing node a, b 6:2 type communication ring Ring, c 4:4 type communication ring Ring, to maximize the use of bandwidth. Wherein, a 7:1 type communication ring Ring is to connect the 8 GPUs in the computing node in a certain logic to form a ring. For example, in combination with the link topology between the 8 GPUs in the computing node shown in Figure 2, a 7:1 type communication ring Ring (see Figure 3a) is: GPU0→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5, wherein GPU0, GPU5 are the starting GPU and the ending GPU of the 7:1 type communication ring respectively, in the communication ring each GPU is connected with a GPU on the left and right respectively, and only receives data from the left GPU and sends data to the right GPU, so it can be seen that when using the given 7:1 type communication ring for data transmission processing, the data needs to pass through 7 hops inside the computing node. It should be noted here that the link topology between the GPUs in the computing node shown in Figure 2 is only exemplary, and in actual application, communication links can exist between the GPUs in the computing node.

[0042] A 6:2 type communication ring Ring is to connect the 7 GPUs in the computing node in a certain logic to form a ring. For example, see Figure 4a, a 6:2 type communication ring Ring shown in the computing node node0 is: GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU3, wherein GPU5, GPU0 are the starting GPU and the ending GPU of the 6:2 type communication ring respectively, in the communication ring each GPU is connected with a GPU on the left and right respectively, and only receives data from the left GPU and sends data to the right GPU, so it can be seen that when using the given 6:2 type communication ring for data transmission processing, the data needs to pass through 6 hops inside the computing node node0.

[0043] A 4:4 type communication ring Ring is formed by connecting 5 GPUs in a computing node in a ring shape according to a certain logic. For example, as shown in FIG. 5a, a 4:4 type communication ring Ring in the computing node node0 is: GPU0→GPU3→GPU2→GPU1→GPU4, where GPU0 and GPU5 are the start GPU and the end GPU of the 4:4 type communication ring respectively, and each GPU in the communication ring is connected with one GPU on the left and on the right respectively, and only receives data from the left GPU and sends data to the right GPU. Therefore, when using the given 4:4 type communication ring to perform data transmission processing, the data needs to pass through 4 hops inside the computing node node0.

[0044] As can be seen from the above examples, the ring type information of each type of communication ring can reflect the number of processors (such as GPUs) in the worker node (such as the computing node) that need to be used to build the type of communication ring, and the number of hops that need to be passed through when performing data transmission. Specifically, the ring type information of a certain type of communication ring contains a ratio, the ratio = first value:second value (for example, the ratio contained in the ring type information of the aforementioned 7:1 type communication ring is 7:1), then: the number of processors that need to be used to build the certain type of communication ring can be equal to the first value corresponding to the certain type of communication ring plus one; and when using the type of communication ring to perform data transmission processing, the number of hops that the data needs to pass through is the first value corresponding to the type of communication ring, for example, when using the given 7:1 type communication ring, 6:2 type communication ring, and 4:4 type communication ring to perform data transmission processing, the data needs to pass through 7 hops, 6 hops, and 4 hops respectively inside the corresponding computing node.

[0045] The ring type information of each type of communication ring can reflect, in addition to the number of hops required for data transmission by the type of communication ring and the number of processors in the worker node required for building the type of communication ring, other information, such as bandwidth occupation information corresponding to data transmission by the type of communication ring. Specifically, the bandwidth occupation information includes occupied intra-node bandwidth and occupied inter-node network bandwidth. The occupied intra-node bandwidth can be equal to the first number corresponding to the type of communication ring multiplied by the communication ring bandwidth of the type of communication ring. The occupied inter-node network bandwidth can be equal to the second number corresponding to the type of communication ring multiplied by the communication ring bandwidth of the type of communication ring. The communication ring bandwidth of different types of communication rings can be set to be the same, and specifically, the communication ring bandwidth can be denoted as x. Based on the above content, for example, taking a 7:1 type communication ring as an example, the ring type information of the 7:1 type communication ring includes the ratio of 7:1. The 7 (as the first number) in the ratio of 7:1 can reflect that data needs to pass through 7 hops in the computing node when using one type of communication ring for data transmission, and the corresponding intra-node bandwidth needs to be occupied as 7*x. The 1 (as the second number) in the ratio of 7:1 can reflect that the corresponding inter-node network bandwidth needs to be occupied as 1*x when using one type of communication ring for data transmission. Similarly, for the ring type information of a 6:2 type communication ring, the ratio of 6:2 includes the ratio of 6:2. The 6 (as the first number) in the ratio of 6:2 can reflect that data needs to pass through 6 hops in the computing node when using one type of communication ring for data transmission, and the corresponding intra-node bandwidth needs to be occupied as 6*x, etc. The 2 (as the second number) in the ratio of 6:2 can reflect that the corresponding inter-node network bandwidth needs to be occupied as 2*x when using one type of communication ring for data transmission. For the ring type information of a 4:4 type communication ring, the ratio of 4:4 includes the ratio of 4:4. The first 4 (as the first number) in the ratio of 4:4 can reflect that data needs to pass through 4 hops in the computing node when using one type of communication ring for data transmission, and the corresponding intra-node bandwidth needs to be occupied as 4*x, etc. The second 4 (as the second number) in the ratio of 4:4 can reflect that the corresponding inter-node network bandwidth needs to be occupied as 4*x when using one type of communication ring for data transmission.

[0046] 12) Establish constraint information: assuming that the communication ring bandwidth of each generated communication ring Ring is x, then: the total intra-node bandwidth constraint of the computing node is, for example, the NVLink total bandwidth constraint is: (a*7+b*6+c*4)x<e; the total inter-node network bandwidth constraint between two computing nodes is: (a*1+b*2+c*4)x<d; the bandwidth uniform distribution constraint (aiming to avoid cross-machine communication bottlenecks) is: (a+b*2+c*4)%8=0; the objective function is: maximize: (a+b+c)*x.

[0047] Based on the above problem formalization, the values of a, b and c can be solved by writing a corresponding program. That is, the number of different types of communication rings Ring that need to be generated in each worker node can be obtained. Wherein, the 8 appearing above represents the total number of processors (such as GPU) in the computing node.

[0048] It should be noted that: the types of communication rings that need to be generated in the computing node can be determined according to user input information, or can be obtained from the locally stored node configuration information, which is not limited here. A large number of experimental studies have shown that when there are 8 GPUs in the computing node, generating the three types of communication rings (7:1, 6:2, 4:4) given in the foregoing example in the computing node can make the ratio of the node internal bandwidth and the inter-node network bandwidth used by the algorithm closer to the actual ratio of the node internal bandwidth and the inter-node network bandwidth in the network.

[0049] After determining the number of various types of communication rings that need to be generated in the computing node, the specific implementation of constructing the corresponding number of various types of communication rings in the computing node will be described in detail in other embodiments below, and will not be described here.

[0050] 2, uniform distribution of bandwidth.

[0051] Since multiple communication rings need to be generated in a single computing node in the network system and the multiple communication rings have multiple types, bandwidth resources can be fully utilized. The communication rings between different computing nodes need to be connected to realize cross-machine communication. In order to avoid cross-machine communication bottlenecks, the inter-machine bandwidth can be uniformly distributed among different communication rings.

[0052] For example, a 7:1 type communication ring is formed by connecting GPU0, GPU3, GPU2, GPU1, GPU4, GPU7, GPU6 and GPU5 in a ring. Since each communication ring Ring requires an input in / output out network card, in order to ensure uniform distribution of network card bandwidth, only the number of generated 7:1 type communication rings Ring needs to be a multiple of the number of network cards in the computing node, and these 7:1 type communication rings Ring use different In / out network cards. In addition, each communication ring uses 7 NVLink links (this is described in the simplex case, if the duplex case, 14 NVLink links will be used), since multiple GPUs in the computing node communicate through the corresponding switch chip (Switch), that is, the computing node topology communication is GPU←→Switch←→NIC (network card), so under the condition of uniform distribution of network card bandwidth, the bandwidth in the computing node will also be uniformly distributed. The number of network cards configured in a computing node is generally 1, 2, 4 or 8, assuming that the number of configured network cards is 8, in this case, in order to ensure uniform distribution of network card bandwidth, only the number of generated 7:1 type communication rings needs to be a multiple of 8, and these 7:1 type communication rings are used as much as possible. Uniformly dispersed to different in / out network cards, for example, if the number of generated 7:1 type communication rings is 8, the relationship between the 7:1 type communication ring and the network card can be configured as a one-to-one relationship, that is, different 7:1 type communication rings use different network cards; and if the number of generated 7:1 type communication rings is 16, the relationship between the 7:1 type communication ring and the network card can be configured as a two-to-one relationship, that is, two 7:1 type communication rings share a network card. Similarly, the user can also specify the number of 6:2 type and 4:4 type communication rings Ring and the network card used by each communication ring Ring through interaction, so as to ensure uniform distribution of bandwidth.

[0053] 3. Connection of communication rings between computing nodes.

[0054] In the implementation of the communication ring connection of different computing nodes, the parity of the total number of computing nodes and the parity of the node number of each computing node can be combined to realize the communication between the same numbered GPUs in different computing nodes to the greatest extent, thereby avoiding the communication bottleneck across the machine.

[0055] For example, taking the 7:1 type communication ring as an example, it is assumed that the network system includes a plurality of computing nodes, wherein, when the plurality of computing nodes are grouped into a ring, based on the link topology between the 8 GPUs in the computing node (as shown in FIG. 2), a certain communication ring Ring71 generated in the computing node node0 (the computing node with node number 0) is formed by the GPUs 0, 3, 2, 1, 4, 7, 6 and 5 in the ring, that is, the certain communication ring Ring71 is: GPU0→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5, and the following cases exist.

[0056] Case 1: If the total number of nodes N of the plurality of computing nodes is 2k (k is a positive integer not equal to 0), exemplarily, the connection strategy set for the connection of the communication rings of different computing nodes is: one 7:1 type communication ring used by the computing nodes with odd node numbers is: GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU3→GPU0; one 7:1 type communication ring used by the computing nodes with even node numbers is: GPU0→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5. That is, one 7:1 type communication ring used by all computing nodes with odd node numbers is the same, and one 7:1 type communication ring used by all computing nodes with even node numbers is the same. After connecting the front and back of the aforementioned one 7:1 type communication ring used by different computing nodes, a cross-node communication ring of the network system is also constructed.

[0057] Wherein, the node number of the computing node can be but is not limited to being sequentially increased by one starting from 0, that is, the computing node with node number n=2k-1 is the last computing node (which can be understood as the termination node of the communication ring between the computing nodes), and 2k-1 is an odd number. In the cross-node communication ring, the plurality of computing nodes are arranged in communication according to the increasing node number.

[0058] The understanding of the above case 1 content can also be seen from the example of connecting the front and back of one 7:1 type communication ring in each computing node to form a cross-node communication ring of the cross-computing node shown in FIG. 3a, which is an example with the total number of nodes of the plurality of computing nodes in the network system being 4 (i.e., k=2). As can be seen from this example, communication between computing nodes only occurs between GPUs with the same number.

[0059] Case 2: If the total number of nodes N = 2k+1, exemplary, the connection strategy set when connecting the communication ring between different computing nodes is: a 7:1 type communication ring Ring that the computing node with node number n < 2k-1 and odd needs to use is GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU3→GPU0; a 7:1 type communication ring that the computing node with node number n < 2k and even needs to use is GPU0→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5; and a 7:1 type communication ring Ring that the computing node with node number n = 2k-1 (odd) needs to use is GPU5→...x (x≠GPU0, GPU5), that is, a 7:1 structure type communication ring Ring with starting GPU being GPU5 and ending GPU not being GPU0 and GPU5, specifically, the 7:1 type communication ring Ring can be: GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU0→GPU3; and a 7:1 type communication ring Ring that the computing node with node number n = 2k (even) needs to use is x→...→GPU5, specifically, the 7:1 type communication ring Ring can be: GPU3→GPU0→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5. Thus, except for the last two computing nodes, a 7:1 type communication ring Ring that all computing nodes with odd node numbers need to use is the same, a 7:1 type logical Ring that all computing nodes with even node numbers need to use is the same; a 7:1 type communication ring Ring that the last two computing nodes need to use is different from that of other computing nodes.

[0060] The understanding of the above case 2 content can be seen from the example of connecting one 7:1 type communication ring in each computing node to form a cross-node communication ring across the working nodes shown in FIG. 3b, which is illustrated by taking the total number of nodes of the plurality of computing nodes in the network system as N = 5 (i.e., k = 2). As can be seen from this example, except that the communication between the computing node node4 and the computing node node0 occurs between different numbered GPUs, the communication between other computing nodes only occurs between GPUs with the same number. Since the performance of the same numbered GPUs of different computing nodes is often the same and consistent, making the communication between computing nodes only occur between GPUs with the same number is beneficial to guarantee the efficient node-to-node communication rate, thereby effectively avoiding the cross-machine communication bottleneck.

[0061] From the above, when 8 communication rings of the 7:1 type are constructed in the computing node node0, for the 8 communication rings of the 7:1 type in the computing node node0, when the communication ring connection between computing nodes is performed, for each communication ring of the 7:1 type, a corresponding GPUx can be found, so that the corresponding cross-node communication ring constructed satisfies the cross-machine communication avoidance.

[0062] Similarly, the connection strategy between the 6:2 type communication ring and the 4:4 type communication ring of different computing nodes can also be set respectively.

[0063] In summary, from the above 1-3, it can be seen that the new heterogeneous Ring algorithm provided by the disclosed scheme can automatically generate a heterogeneous communication ring with better performance for a network system architecture, and can automatically optimize the heterogeneous communication ring Ring structure of the network system according to the actual bandwidth ratio between computing nodes and within computing nodes in the network system, and effectively utilize the available bandwidth. Compared with the existing Ring algorithm scheme, the disclosed scheme mathematically formalizes the communication ring generation problem within the computing node, thereby establishing the constraint information of the communication ring generation within the computing node, and using programming to solve, so as to automatically determine the type (such as 7:1 type, 6:2 type, 4:4 type) of the communication ring Ring generated within the computing node and the number of different types of communication rings, to ensure that the bandwidth ratio between computing nodes and within computing nodes used by the communication ring is close to the actual hardware bandwidth ratio; and when establishing the constraint information, the disclosed scheme ensures that the number of communication rings Ring generated within the computing node satisfies the constraint condition of allowing uniform distribution to different In / Out network cards, avoids the cross-machine communication bottleneck, and ensures the uniform use of the bandwidth within the computing node through the communication ring Ring of a specific structure type. In addition, the disclosed scheme formulates different inter-machine communication ring Ring connection strategies for different types of communication rings within the computing node according to the node label parity of the computing node and the parity of the total number of computing nodes, to ensure that the inter-node communication of the computing node only occurs between GPUs with the same number, and to avoid inefficient cross-machine communication.

[0064] Based on the above, the disclosed scheme provides the following embodiments of the communication ring generation method of the network system.

[0065] The execution subject of each method below can be a control node in the network, which can be a separate node independent of multiple working nodes in the network or can also be one of the multiple working nodes in the network, and the embodiment does not limit this. The working node can be understood as the aforementioned computing node.

[0066] FIG. 6 shows a flowchart of a method for generating a communication ring according to an embodiment of the present disclosure. As shown in FIG. 6, the method for generating a communication ring includes steps 101-104.

[0067] 101. Obtain ring type information of various types of communication rings to be created in each working node in a network; wherein the ring type information of a type of communication ring can reflect the number of processors to be used when creating the type of communication ring, and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information including a first value representing the occupation of bandwidth in a node and a second value representing the occupation of bandwidth between nodes in a network; the network includes a plurality of working nodes.

[0068] 102. Determine the number of various types of communication rings to be generated in each working node according to the bandwidth occupation information corresponding to various types of communication rings, the working node information of the network, and the set communication ring bandwidth, and the parameter representing the number of various types of communication rings to be generated in each working node.

[0069] 103. Based on the number of processors to be used when creating various types of communication rings and the number of various types of communication rings to be generated in each working node, select an adaptive number of processors from a plurality of processors in a working node for ring connection operation to generate a plurality of node-level communication rings in the plurality of working nodes, respectively.

[0070] 104. Connect a plurality of node-level communication rings between the plurality of working nodes to construct a plurality of cross-node communication rings of the network; wherein the total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings, and a cross-node communication ring is formed by selecting one node-level communication ring of the same type from the plurality of working nodes and sequentially connecting the selected node-level communication rings end to end.

[0071] In 101, the network can be a network system for performing parallel data processing tasks, such as a network system for distributed model training (as shown in FIG. 1). The network includes a control node (not shown in the figure) and a plurality of working nodes. The working node can be a computing node with data computing and processing capability, such as a server. For details of the plurality of working nodes and their communication, please refer to the relevant content in other embodiments of the present disclosure.

[0072] In addition, when creating a communication ring in a working node, at least part of the plurality of processors in the working node are connected in a ring to form a corresponding communication ring. The plurality of processors in the working node can be CPUs or GPUs, etc. In the distributed training scenario, the plurality of processors in the working node in the present embodiment are GPUs.

[0073] In some examples, in addition to the worker nodes and the control nodes, the network can also include client devices through which users can configure or input some information to interact with the control nodes.

[0074] For example, through an interactive interface provided by the client device, the user can input the following information for the communication ring types to be created in the worker nodes: 7:1, 6:2, 4:4, which can indicate that the user instructs to create the following three types of communication rings in each worker node: a 7:1 type communication ring, a 6:2 type communication ring, and a 4:4 type communication ring. Among them, the 7:1 type communication ring is formed by using 8 processors such as GPUs in the worker node to form a ring structure according to a certain logic group, and the ring type information of the 7:1 type communication ring includes the ratio of 7:1. For information content that can be reflected by the ratio of 7:1, please refer to the related content described in other embodiments of the present disclosure, which will not be described in detail here.

[0075] It should be noted that: in the above example, the 7:1, 6:2, and 4:4 information input by the user can be but not limited to input by clicking, keyboard, voice, etc. For example: the interactive interface of the client device displays a list of communication ring structure types, and the list of communication ring types displays the following list items: 7:1, 6:2, 5:3, 4:4, 3:5, 3:1, etc. The user can click on the three list items of 7:1, 6:2, and 4:4 to input 7:1, 6:2, and 4:4.

[0076] In addition, the ring type information of the multiple types of communication rings to be created in the worker node can also be obtained in other ways, such as from the configuration information of the worker node.

[0077] From the above, the step 101 of "obtaining ring type information of multiple types of communication rings to be created in the worker node in the network" can include: in response to the input operation of the user, obtaining the ring type information of multiple types of communication rings to be created in the worker node from the input information of the user; or, obtaining the ring type information of multiple types of communication rings to be created in the worker node from the locally stored node configuration information. The ring type information of a type of communication ring can reflect the number of processors to be used when creating the type of communication ring and the bandwidth occupation information when transmitting data through the type of communication ring. The bandwidth occupation information includes a first value representing the occupation of the bandwidth in the node and a second value representing the occupation of the network bandwidth between nodes. How the ring type information of a type of communication ring can reflect the bandwidth occupation information described above can be referred to the related content described for the 7:1 type communication ring in other embodiments of the present disclosure, which will not be described in detail here.

[0078] And, in the above 102, the constraint information and the objective function can be determined according to the bandwidth occupation information, the working node information of the network, and the set communication ring bandwidth and the parameter representing the number of various types of communication rings to be generated in each working node, and then the objective function is solved based on the constraint information to obtain the number of various types of communication rings to be generated in each working node.

[0079] Based on this, in a specific example, the above 102 can be implemented by the following steps.

[0080] 1021. Determine the constraint information according to the bandwidth occupation information corresponding to various types of communication rings, the working node information, and the parameter representing the number of various types of communication rings to be generated in each working node.

[0081] 1022. Determine the objective function according to the parameter representing the number of various types of communication rings to be generated in each working node and the communication ring bandwidth.

[0082] 1023. Solve the objective function based on the constraint information to obtain the number of various types of communication rings to be generated in each working node.

[0083] In the above 1021-1023, the working node information of the network includes the total bandwidth of a single working node, the number of network cards in the node, the total network bandwidth between two working nodes, etc.

[0084] The communication ring bandwidth (denoted as x) can be set by the user through the client device according to the user's own needs, or a default value can also be used, and the present embodiment does not limit this. In the present embodiment, the communication ring bandwidth of all types of communication rings is the same by default, but it can also be different.

[0085] In addition, taking the three types of communication rings of 7:1 type communication ring, 6:2 type communication ring and 4:4 type communication ring to be created in each working node as an example, the number of 7:1 type communication rings to be generated in the working node, the number of 6:2 type communication rings to be generated in the working node, and the number of 4:4 type communication rings to be generated in the working node can be represented by parameters a, b and c respectively.

[0086] The determined constraint information includes: a node internal bandwidth constraint, an inter-node network bandwidth constraint, and a bandwidth uniform distribution constraint. The node internal bandwidth constraint is established based on the parameters representing the number of various types of communication rings to be generated in each working node, the first numerical values corresponding to the various types of communication rings, the communication ring bandwidths, and the total bandwidth in the node. The inter-node network bandwidth constraint is established based on the parameters representing the number of various types of communication rings to be generated in each working node, the second numerical values corresponding to the various types of communication rings, the communication ring bandwidths, and the total inter-node network bandwidth. The bandwidth uniform distribution constraint is established based on the parameters representing the number of various types of communication rings to be generated in each working node, the second numerical values corresponding to the various types of communication rings, and the number of network cards in the node.

[0087] For example, as given in the example of the foregoing other embodiments, the node internal bandwidth constraint expression can be: (a*7+b*6+c*4)x<e, the inter-node network bandwidth constraint expression can be: (a*1+b*2+c*4)x<d, and the bandwidth uniform distribution constraint expression can be: (a+b*2+c*4)%8=0, where 8 represents the number of network cards (i.e., the number of internal networks) in the working node.

[0088] In addition, the determined objective function is to maximize the product value between the target sum value and the communication ring bandwidth x, where the target sum value is equal to the sum value between the parameters representing the number of various types of communication rings to be generated in each working node. As an example, the expression of the objective function can be: maximize: (a+b+c)*x.

[0089] The above 1023, the constraint information based on the solution of the objective function can be realized by programming, so as to obtain the number of various types of communication rings to be generated in each working node. As an example, the specific values of the parameters a, b and c are obtained, so that the number of 7:1 type communication rings (such as a=8), the number of 6:2 type communication rings (such as b=16), and the number of 4:4 type communication rings (such as c=32) to be constructed in the working node can be obtained.

[0090] In the above 103, it can be understood that the node-level communication ring described herein is the communication ring generated in the working node described in the other embodiments of the present disclosure. As shown in FIG. 3a, one 7:1 type communication ring “GPU0→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5” in the computing node node0 is a node-level communication ring.

[0091] The plurality of node-level communication rings generated in the worker node has multiple types of communication rings. Assuming that the plurality of node-level communication rings needed to be generated in each worker node includes 8 7:1 type communication rings, 16 6:1 type communication rings, and 32 4:4 type communication rings, then, taking the generation of the plurality of node-level communication rings in a first worker node (as one of the plurality of worker nodes) as an example, the generation process can be as follows: the total number of processors (such as GPUs) included in the first worker node is 8, and for the first worker node, 8 GPUs in the first worker node can be used to be logically combined to form 8 node-level communication rings (all 7:1 type communication rings) in the first worker node. Similarly, a corresponding number of node-level communication rings of other types can also be formed in the first worker node, specifically: 7 GUPs can be selected from the 8 GPUs included in the first worker node to be logically combined to form 16 node-level communication rings (all 6:2 type communication rings) in the first worker node; and 5 GUPs can be selected from the 8 GPUs included in the first worker node to be logically combined to form 32 node-level communication rings (all 4:4 type communication rings) in the first worker node. When selecting GPUs, the GPUs can be randomly selected or can be selected based on information such as the usage rate (also referred to as the occupancy rate) of the GPUs, which is not limited here.

[0092] In the ring, two GPUs can be selected first to be used as a starting GUP and a terminal GUP, and then a corresponding number of GPUs can be selected as intermediate nodes. The selection of the starting GUP and the terminal GUP can be related to the node-level communication ring connection strategy (or the total number of worker nodes) described below.

[0093] For more detailed implementation description of generating a plurality of node-level communication rings (including different types of communication rings) in each worker node, please refer to the related content given below for step 1032, which will not be described in detail here.

[0094] The technical scheme provided by the embodiment is that when a communication ring in each working node in a network is established, the ring type information of various types of communication rings to be created in each working node in the network is acquired first; the ring type information of one type of communication ring can reflect the number of processors to be used when the type of communication ring is created and bandwidth occupation information when data is transmitted through the type of communication ring, and the bandwidth occupation information includes a first value representing the occupied bandwidth in a node and a second value representing the occupied network bandwidth between nodes; the network includes a plurality of working nodes; then, the constraint information and the objective function are determined according to the bandwidth occupation information of various types of communication rings, the working node information of the network, and the set communication ring bandwidth and the parameter representing the number of various types of communication rings to be generated in each working node, so that the number of various types of communication rings to be generated in each working node is obtained by solving the objective function based on the constraint information. In this way, the number of various types of communication rings to be generated is determined, which can effectively ensure that the ratio of the in-node bandwidth used by the communication ring to the inter-node network bandwidth is close to the bandwidth ratio of the actual hardware in the network (i.e., the actual ratio of the inter-working node bandwidth to the in-working node bandwidth), thereby improving the bandwidth utilization. Further, based on the number of processors to be used when creating the various types of communication rings and the number of various types of communication rings to be generated in each working node, the ring-like connection operation is performed on the selected adaptive number of processors from the plurality of processors in each working node to generate a plurality of node-level communication rings in each working node. The plurality of node-level communication rings generated in the working node have various types of communication rings, and the available bandwidth can be effectively utilized through the various types of communication rings.

[0095] It is considered that in order to perform cross-node communication based on a ring, a plurality of node-level communication rings between working nodes need to be connected to form a larger cross-node communication ring. In order to avoid the cross-node communication bottleneck, the connection strategy adopted when connecting the plurality of node-level communication rings between working nodes is to maximize the communication between nodes. In order to make the plurality of node-level communication rings generated in the working node better satisfy this connection strategy, the generation of the node-level communication ring can also be combined with this connection strategy. The connection strategy of various types of communication rings between working nodes can be implemented based on the working node information of the network. Therefore, the above-mentioned 103 “based on the number of processors to be used when creating the various types of communication rings and the number of various types of communication rings to be generated in each working node, performing the ring-like connection operation on the selected adaptive number of processors from the plurality of processors in each working node to generate a plurality of node-level communication rings in each working node” includes steps 1031 to 1032.

[0096] 1031. Determine the connection strategy of various types of communication rings among the plurality of worker nodes according to the worker node information of the network.

[0097] 1032. For each of the plurality of worker nodes, select an adaptive number of processors from the plurality of processors in the worker node for ring connection operation to generate a plurality of node-level communication rings in the worker node based on the connection strategy corresponding to the various types of communication rings, the number of processors required for creation, and the number of processors required for generation in each worker node.

[0098] The worker node information of the network can further include the total number of worker nodes in the network and the node number of each worker node. The node numbers of the plurality of worker nodes in the network can be, but are not limited to, numbered starting from 0 and using a continuous incremental method.

[0099] In determining the connection strategy of various types of communication rings among the plurality of worker nodes, the parity of the total number of worker nodes and the parity of the node number of each worker node can be determined. That is, the first type of communication ring (such as a 7:1 type of communication ring) is one of the various types of communication rings. In a specific implementation, the step 1031 of determining the connection strategy of the first type of communication ring among the plurality of worker nodes according to the worker node information of the network includes the step 10311.

[0100] 10311. Determine the first type of communication ring to be used by each of the plurality of worker nodes when constructing a cross-node communication ring according to the parity of the total number of worker nodes and the parity of the node number of each of the plurality of worker nodes, and connect the first type of communication ring at the beginning and the end.

[0101] In the connection at the beginning and the end, the two processors connected between at least two of the plurality of worker nodes have the same number.

[0102] Specifically, in the cross-node communication ring, the plurality of worker nodes are arranged in a ring according to the increasing node number (see FIG. 3a or FIG. 3b). The specific implementation of the step 10311 can include the following steps S1 to S2.

[0103] S1, if the total number of the working nodes N=2K, K is a positive integer greater than or equal to 1, then all odd-numbered working nodes in the plurality of working nodes need to use the first communication ring in it, and all even-numbered working nodes need to use the second communication ring in it; the first communication ring and the second communication ring both belong to the first type of communication ring, and the termination processor of the first communication ring and the start processor of the second communication ring have the same number, and the start processor of the first communication ring and the termination processor of the second communication ring have the same number.

[0104] S2, if the total number of the working nodes N=2K+1, and the start node number of the working node is zero, then all odd-numbered working nodes with node numbers less than or equal to 2K-2 in the plurality of working nodes need to use the first communication ring in it, and all even-numbered working nodes need to use the second communication ring in it, and the working node with node number equal to 2K-1 needs to use the third communication ring in it, and the working node with node number equal to 2K needs to use the fourth communication ring in it; wherein the start processor of the third communication ring and the termination processor of the second communication ring and the termination processor of the fourth communication ring have the same number, the number of the termination processor of the third communication ring is different from the numbers of the start processor and the termination processor of the second communication ring, but the same as the number of the start processor of the fourth communication ring; the third communication ring and the fourth communication ring both belong to the first type of communication ring.

[0105] For example description of the above step S1, please refer to the content related to "case 1" in the foregoing other embodiments, and also refer to FIG. 3a, FIG. 4a and FIG. 5a. As shown in FIG. 3a, the working node in the embodiment is the computing node in FIG. 3a, when constructing a cross-node communication ring, all even-numbered computing nodes (such as the computing node with node number 0, node0, and the computing node with node number 2, node2) need to use the first communication ring GPU0→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5, and all odd-numbered computing nodes (such as the computing node with node number 1, node1, and the computing node with node number 3, node3) need to use the second communication ring GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU3→GPU0.

[0106] For example description of the above step S2, please refer to the content related to "case 2" described in the foregoing other embodiments and also refer to FIG. 3b, FIG. 4b and FIG. 5b, which will not be described in detail here.

[0107] It should be noted that the connection strategy determination of other types of communication rings among multiple worker nodes can refer to the connection strategy determination described in the foregoing steps S1-S2 for the first type of communication ring, which will not be described here.

[0108] In 1032, in combination with the content given for step 1031, if the first type of communication ring is one of multiple types of communication rings, for example, the first type of communication ring is one of 7:1 type communication ring, 6:2 type communication ring, and 4:4 type communication ring, in order to meet the connection strategy corresponding to the first type of communication ring, the selection of the start processor and the end processor when creating the corresponding number of first type of communication rings in each worker node needs to meet certain requirements. After determining the selected start processor and end processor according to this requirement, a number of other processors can be selected as intermediate processors based on the number of processors needed to create the first type of communication ring, so as to realize the creation of the first type of communication ring (node-level communication ring) based on the selected start processor, end processor and intermediate processor. Among them, the number of intermediate processors = the number of processors needed to create the first type of communication ring - 2.

[0109] In specific implementation, assuming that the total number of processors (such as GPU) in each worker node is m, the m processors include: processor 0, processor 1, processor 2,..., processor m-1, then: A, in the scenario where the total number of worker nodes of the network (i.e. the total number of multiple worker nodes) N is even (N = 2K), based on the communication ring connection strategy between worker nodes described in the foregoing step S1, it can be known that when creating the first type of communication ring in each worker node, it needs to be ensured that the same processor is selected as the start processor and the end processor in the used worker node, wherein, here "the same processor" refers to the same number of processors, for example, the two processors with number 0 and number 5 can be selected as the start processor and the end processor in all worker nodes; in addition, if a selected processor is used as a start processor in an even node, it will be used as an end processor in an odd node; conversely, if a selected processor is used as an end processor in an even node, it will be used as a start processor in an odd node.

[0110] Based on the above content, in this scenario, if one form of the first type of communication ring created in all even nodes is: processor i→.......→processor j, then correspondingly, the first type of communication ring in the form of: processor j→.......→processor i needs to be created in all odd nodes. Wherein, processor i and processor j are any two different processors of the m processors.

[0111] B、In the scenario where the total number of working nodes N of the network is odd (N = 2K + 1), based on the working node intercommunication ring connection strategy described in the foregoing step S2, it can be known that when the first type of communication ring is created in each working node, it needs to be ensured that: assuming that the multiple working nodes are numbered in a self-incrementing manner starting from zero, then, for all working nodes with node numbers less than or equal to 2K-2, the same processor is selected to be used as the starting processor and the ending processor, where the "same processor" refers to the same processor number, for example, the two processors with numbers 0 and 5 can be selected to be used as the starting processor and the ending processor in all working nodes with node numbers less than or equal to 2K-2. In addition, for all working nodes with node numbers less than or equal to 2K-2, if one selected processor is used as the starting processor in an even node, it is used as the ending processor in an odd node; conversely, if one selected processor is used as the ending processor in an even node, it is used as the starting processor in an odd node.

[0112] For the working node with node number 2K-1, the number of the processor selected to be used as the starting processor is the same as that of the processor selected to be used as the ending processor in the foregoing even node with node number less than or equal to 2K-2, and the number of the processor selected to be used as the ending processor is different from the numbers of the two processors selected to be used as the starting processor and the ending processor in the foregoing working node with node number less than or equal to 2K-2. In addition, if one selected processor is used as the starting processor in the working node with node number 2K-1, it is used as the ending processor in the working node with node number 2K; conversely, if one selected processor is used as the ending processor in the working node with node number 2K-1, it is used as the starting processor in the working node with node number 2K.

[0113] Based on the foregoing, in this scenario, if the first type of communication ring created in all even nodes with node numbers less than or equal to 2K-2 is in the form of processor i→.......→processor j, then correspondingly, the first type of communication ring needs to be created in all odd nodes with node numbers less than or equal to 2K-2 in the form of processor j→.......→processor i; the first type of communication ring needs to be created in the working node with node number 2K-1 in the form of processor j→.......→processor x; and the first type of communication ring needs to be created in the working node with node number 2K in the form of processor x→.......→processor j. Wherein, the processor i, the processor j and the processor x are any three different processors in the m processors.

[0114] To facilitate the understanding of the foregoing step 1032, specific examples are listed below for illustration.

[0115] Assuming that each of the plurality of worker nodes in the network includes eight GPUs, GPU0, GPU1, GPU2, GPU3, GPU4, GPU5, GPU6 and GPU7, and that the plurality of node-level communication rings to be generated in each worker node includes eight 7:1 type communication rings, sixteen 6:1 type communication rings and thirty-two 4:4 type communication rings, the plurality of worker nodes are numbered in a self-incremental manner starting from zero. Then: 1) when eight 7:1 type communication rings are created in a worker node, based on the ring type information of this type of communication ring such as the ratio of 7:1, it can be known that eight GPUs (i.e. all GPUs) in the worker node need to be used.

[0116] (a11) In combination with FIG. 3a, in the scenario that the network comprises four worker nodes: the computing node nodeO, the computing node nodel, the computing node node2 and the computing node node3, and in the case that eight 7: 1 type communication rings are created in the computing node nodeO, exemplary, GPUO and GPU5 can be selected randomly from the eight GUPs comprised in the computing node nodeO as the start GPU and the end GPU respectively, and the remaining GPUs 1, 2, 3, 4, 6 and 7 are used as intermediate GPUs. That is, one form of the 7: 1 type communication ring created in the computing node nodeO can be: GPUO→......→GPU5, wherein the intermediate GPUs between GPUO and GPU5 are the remaining six GPUs and the order of the remaining six GPUs is not limited. From the above, the eight 7: 1 type communication rings created in the computing node nodeO can all be in the form of GPUO→......→GPU5, for example, the eight 7: 1 type communication rings created can be, but are not limited to, the following Rin71~Ring78: Ring71: GPUO→GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5; Ring72: GPUO→GPU2→GPU3→GPU1→GPU4→GPU7→GPU6→GPU5; Ring73: GPUO→GPU2→GPU1→GPU3→GPU4→GPU7→GPU6→GPU5; Ring74: GPUO→GPU2→GPU1→GPU4→GPU3→GPU7→GPU6→GPU5; Ring75: GPUO→GPU2→GPU1→GPU4→GPU7→GPU3→GPU6→GPU5; Ring76: GPUO→GPU2→GPU1→GPU4→GPU7→GPU6→GPU3→GPU5; Ring77: GPUO→GPU1→GPU2→GPU4→GPU7→GPU6→GPU3→GPU5; Ring78: GPUO→GPU1→GPU4→GPU2→GPU7→GPU6→GPU3→GPU5.

[0117] In FIG. 3a, only the Ring71 created above is shown in the computing node nodeO.

[0118] and correspondingly, when creating 8 7: 1 type communication rings in the computing node node1, GPU5 and GPU0 are selected from the 8 GUPs contained in the computing node node1 as the start GPU and the end GPU respectively, and the remaining GPU1, GPU2, GPU3, GPU4, GPU6 and GPU7 are selected as intermediate GPUs. That is, one form of the 7: 1 type communication ring created in the computing node node1 can be: GPU5→……→GPU0, wherein the intermediate GPUs between GPU5 and GPU0 are the remaining 6 GPUs and there is no limitation on the arrangement order of the remaining 6 GPUs. From the above, the 8 7: 1 type communication rings created in the computing node node1 can all be in the form of GPU0→……→GPU5, for example, the 8 7: 1 type communication rings created can be but not limited to the following Ring79~Ring716 shown as follows: Ring79: GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU3→GPU0; Ring710: GPU5→GPU7→GPU6→GPU4→GPU1→GPU2→GPU3→GPU0; Ring711: GPU5→GPU7→GPU4→GPU6→GPU1→GPU2→GPU3→GPU0; Ring712: GPU5→GPU7→GPU4→GPU1→GPU6→GPU2→GPU3→GPU0; Ring713: GPU5→GPU7→GPU4→GPU1→GPU2→GPU6→GPU3→GPU0; Ring714: GPU5→GPU7→GPU4→GPU1→GPU2→GPU3→GPU6→GPU0; Ring715: GPU5→GPU4→GPU7→GPU1→GPU2→GPU3→GPU6→GPU0; Ring716: GPU5→GPU4→GPU1→GPU7→GPU2→GPU3→GPU6→GPU0.

[0119] In FIG. 3a, only the 7: 1 type communication ring Ring79 created above is shown in the computing node node1.

[0120] For creating 8 7: 1 type communication rings in the computing node node2, please refer to the above-mentioned related content of creating 8 7: 1 type communication rings in the computing node node0, and for creating 8 7: 1 type communication rings in the computing node node3, please refer to the above-mentioned related content of creating 8 7: 1 type communication rings in the computing node node1.

[0121] (b11) In combination with FIG. 3b, in the scenario that the network collectively comprises five worker nodes of the computing node node0, the computing node node1, the computing node node2, the computing node node3, and the computing node node4, the implementation of creating 8 communication rings of the 7:1 type in the computing node node0, the computing node node1, and the computing node node2 respectively can be seen from the related content given in the foregoing (a11), which will not be described in detail here. Thus, for example, the 8 communication rings of the 7:1 type created in the computing node node0 and the computing node node2 are all in the form of GPU0→GPU5→GPU5→GPU0, such as Ring71~Ring78 given in the foregoing (a11); and the 8 communication rings of the 7:1 type created in the computing node node1 are all in the form of GPU5→GPU0→GPU0→GPU5, such as Ring79~Ring716 given in the foregoing (a11).

[0122] Based on the above example, when the 8 7:1 type communication rings are created in the computing node node3, GPU5 can be selected as the start GPU from the 8 GUPs included in the computing node node3, and one GPU can be selected as the end GPU from the other GPUs except GPU5 and GPU0, such as selecting GPU3 as the end GPU, and the remaining GPU0, GPU1, GPU2, GPU4, GPU6 and GPU7 as intermediate GPUs. That is, one form of the 7:1 type communication ring created in the computing node node3 can be: GPU5→……→GPU3, wherein the intermediate GPUs between GPU5 and GPU3 are the remaining 6 GPUs and there is no limitation on the arrangement order of the remaining 6 GPUs. For example, the 8 7:1 type communication rings created in the computing node node3 can be, but are not limited to, the following Ring717 to Ring724 shown: Ring717: GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU0→GPU3; Ring718: GPU5→GPU7→GPU6→GPU4→GPU1→GPU2→GPU0→GPU3; Ring719: GPU5→GPU7→GPU4→GPU6→GPU1→GPU2→GPU0→GPU3; Ring720: GPU5→GPU7→GPU4→GPU1→GPU6→GPU2→GPU0→GPU3; Ring721: GPU5→GPU7→GPU4→GPU1→GPU2→GPU6→GPU0→GPU3; Ring722: GPU5→GPU7→GPU4→GPU1→GPU2→GPU0→GPU6→GPU3; Ring723: GPU5→GPU4→GPU7→GPU1→GPU2→GPU0→GPU6→GPU3; Ring724: GPU5→GPU4→GPU1→GPU7→GPU2→GPU0→GPU6→GPU3.

[0123] In FIG. 3b, only the Ring717 of the above created 7:1 type communication ring is shown in the computing node node3.

[0124] and correspondingly, when creating 8 7: 1 type communication rings in the computing node node4, GPU3 and GPU5 can be selected as the start GPU and the end GPU respectively from the 8 GUPs contained in the computing node node4, and the remaining GPU0, GPU1, GPU2, GPU4, GPU6 and GPU7 are used as intermediate GPUs. That is, one form of the 7: 1 type communication ring created in the computing node node4 can be: GPU3→......→GPU5, wherein the intermediate GPUs between GPU3 and GPU5 are the remaining 6 GPUs and there is no limitation on the arrangement order of the remaining 6 GPUs. For example, the 8 7: 1 type communication rings created in the computing node node4 can be, but are not limited to, the following Ring725~Ring732: Ring725: GPU3→GPU0→GPU2→GPU1→GPU4→GPU7→GPU6→GPU5; Ring726: GPU3→GPU2→GPU0→GPU1→GPU4→GPU7→GPU6→GPU5; Ring727: GPU3→GPU2→GPU1→GPU0→GPU4→GPU7→GPU6→GPU5; Ring728: GPU3→GPU2→GPU1→GPU4→GPU0→GPU7→GPU6→GPU5; Ring729: GPU3→GPU2→GPU1→GPU4→GPU7→GPU0→GPU6→GPU5; Ring730: GPU3→GPU2→GPU1→GPU4→GPU7→GPU6→GPU0→GPU5; Ring731: GPU3→GPU1→GPU2→GPU4→GPU7→GPU6→GPU0→GPU5; Ring732: GPU3→GPU1→GPU4→GPU2→GPU7→GPU6→GPU0→GPU5.

[0125] In FIG. 3b, only the Ring725 created above is shown in the computing node node4.

[0126] 2) When creating 16 6: 2 type communication rings in the worker node, based on the ring type information of this type of communication ring such as the ratio of 6: 2, it can be known that 7 GPUs in the worker node need to be used.

[0127] (a12) In combination with FIG. 4a, in the scenario that the network collectively comprises four worker nodes of the computing node node0, the computing node node1, the computing node node2 and the computing node node3, and 16 6:2 type communication rings are created in the computing node node0, exemplary, GPU5 and GPU3 can be first randomly selected from the 8 GUPs comprised in the computing node node0 as a starting GPU and a terminal GPU respectively, and then 5 GPUs are randomly selected from the remaining 6 GPUs of GPU0, GPU1, GPU2, GPU4, GPU6 and GPU7 as intermediate GPUs, for example, GPU6, GPU7, GPU4, GPU1 and GPU2 are selected as the intermediate GPUs. That is, one form of the 6:2 type communication ring created in the computing node node0 can be: GPU5→……→GPU3, wherein the arrangement order of the 5 different GPUs (such as GPU6, GPU7, GPU4, GPU1 and GPU2) located between GPU5 and GPU3 is not limited.From the above, the 16 6:2 type communication rings created in the computing node node0 can all be in the form of GPU5→...→GPU3, for example, the 16 6:2 type communication rings created in the computing node node0 can be, but are not limited to, the following Ring61~Ring616: Ring61: GPU5→GPU6→GPU7→GPU4→GPU1→GPU2→GPU3; Ring62: GPU5→GPU7→GPU6→GPU4→GPU1→GPU2→GPU3; Ring63: GPU5→GPU7→GPU4→GPU6→GPU1→GPU2→GPU3; Ring64: GPU5→GPU7→GPU4→GPU1→GPU6→GPU2→GPU3; Ring65: GPU5→GPU7→GPU4→GPU1→GPU2→GPU6→GPU3; Ring66: GPU5→GPU4→GPU7→GPU1→GPU2→GPU6→GPU3; Ring67: GPU5→GPU4→GPU1→GPU7→GPU2→GPU6→GPU3; Ring68: GPU5→GPU4→GPU1→GPU2→GPU7→GPU6→GPU3; Ring69: GPU5→GPU4→GPU1→GPU2→GPU6→GPU7→GPU3; Ring610: GPU5→GPU1→GPU4→GPU2→GPU6→GPU7→GPU3; Ring611: GPU5→GPU1→GPU2→GPU4→GPU6→GPU7→GPU3; Ring612: GPU5→GPU1→GPU2→GPU6→GPU4→GPU7→GPU3; Ring613: GPU5→GPU1→GPU2→GPU6→GPU7→GPU4→GPU3; Ring614: GPU5→GPU2→GPU1→GPU6→GPU7→GPU4→GPU3; Ring615: GPU5→GPU2→GPU6→GPU1→GPU7→GPU4→GPU3; Ring616: GPU5→GPU2→GPU6→GPU7→GPU1→GPU4→GPU3.

[0128] In FIG. 4a, only the Ring61 6:2 type communication ring created above is shown in the computing node node0.

[0129] Correspondingly, when creating the 6:2 type communication ring in the computing node node1, GPU3 and GPU5 are selected as the start GPU and the end GPU from the 8 GUPs included in the computing node node1, and then 5 GPUs are randomly selected from the remaining 6 GPUs, namely GPU0, GPU1, GPU2, GPU4, GPU6 and GPU7, as intermediate GPUs, for example, GPU0, GPU2, GPU7, GPU6 and GPU4 are selected as the intermediate GPUs. That is, one form of the 6:2 type communication ring created in the computing node node1 can be: GPU3→……→GPU5, wherein the arrangement order of the 5 GPUs (intermediate GPUs) between GPU3 and GPU5, such as GPU0, GPU2, GPU7, GPU6 and GPU4, is not limited.From the above, the 16 6:2 type communication rings created in the computing node node1 can all be in the form of GPU3→......→GPU5, for example, the 16 6:2 type communication rings created in the computing node node1 can be, but are not limited to, the following Rin617 to Ring632: Ring617: GPU3→GPU0→GPU2→GPU7→GPU6→GPU4→GPU5; Ring618: GPU3→GPU2→GPU0→GPU7→GPU6→GPU4→GPU5; Ring619: GPU3→GPU2→GPU7→GPU0→GPU6→GPU4→GPU5; Ring620: GPU3→GPU2→GPU7→GPU6→GPU0→GPU4→GPU5; Ring621: GPU3→GPU2→GPU7→GPU6→GPU4→GPU0→GPU5; Ring622: GPU3→GPU7→GPU2→GPU6→GPU4→GPU0→GPU5; Ring623: GPU3→GPU7→GPU6→GPU2→GPU4→GPU0→GPU5; Ring624: GPU3→GPU7→GPU6→GPU4→GPU2→GPU0→GPU5; Ring625: GPU3→GPU7→GPU6→GPU4→GPU0→GPU2→GPU5; Ring626: GPU3→GPU6→GPU7→GPU4→GPU0→GPU2→GPU5; Ring627: GPU3→GPU6→GPU4→GPU7→GPU0→GPU2→GPU5; Ring628: GPU3→GPU6→GPU4→GPU0→GPU7→GPU2→GPU5; Ring629: GPU3→GPU6→GPU4→GPU0→GPU2→GPU7→GPU5; Ring630: GPU3→GPU4→GPU6→GPU0→GPU2→GPU7→GPU5; Ring631: GPU3→GPU4→GPU0→GPU6→GPU2→GPU7→GPU5; Ring632: GPU3→GPU4→GPU0→GPU2→GPU6→GPU7→GPU5.

[0130] In FIG. 4a, only the Ring617, which is the 6:2 type communication ring created above, is shown in the computing node node1.

[0131] For creating 16 6:2 type communication rings in the compute node node2, please refer to the foregoing content about creating 16 6:2 type communication rings in the compute node node0, and for creating 16 6:2 type communication rings in the compute node node3, please refer to the foregoing content about creating 16 6:2 type communication rings in the compute node node1.

[0132] (b12) In the network containing five working nodes of the compute node node0, the compute node node1, the compute node node2, the compute node node3 and the compute node node4, the implementation of creating 16 6:2 type communication rings in the compute node node0, the compute node node1 and the compute node node2 respectively can be seen from the foregoing (a12) and will not be described here in detail. Thus, for example, the 16 6:2 type communication rings created in the compute node node0 and the compute node node2 can all be in the form of GPU5→……→GPU3, such as Ring161 to Ring616, and the 16 6:2 type communication rings created in the compute node node1 can all be in the form of GPU3→……→GPU5, such as Ring617 to Ring632.

[0133] Based on the above example, when 16 communication rings of type 6:2 are created in the computing node node3, GPU3 can be selected as the start GPU from the 8 GUPs included in the computing node node3, one GPU can be selected as the end GPU from the other GPUs except GPU3 and GPU5, for example, GPU6 can be selected as the end GPU, and then 5 GPUs can be randomly selected as intermediate GPUs from the remaining GPUs 0, 1, 2, 3, 4 and 7, for example, GPUs 0, 2, 7, 4 and 5 can be selected as the intermediate GPUs. From the above, one form of the 6:2 type communication ring created in the computing node node3 can be: GPU3→......→GPU6, wherein there are 5 GPUs between GPU3 and GPU6 and the arrangement order of the 5 GPUs (as intermediate GPUs) is not limited.Exemplarily, the 16 6:2 type communication rings created in the computing node node3 can be, but are not limited to, the following Rin633~Ring648: Ring633: GPU3→GPU0→GPU2→GPU7→GPU4→GPU5→GPU6; Ring634: GPU3→GPU2→GPU0→GPU7→GPU4→GPU5→GPU6; Ring635: GPU3→GPU2→GPU7→GPU0→GPU4→GPU5→GPU6; Ring636: GPU3→GPU2→GPU7→GPU4→GPU0→GPU5→GPU6; Ring637: GPU3→GPU2→GPU7→GPU4→GPU5→GPU0→GPU6; Ring638: GPU3→GPU7→GPU2→GPU4→GPU5→GPU0→GPU6; Ring639: GPU3→GPU7→GPU4→GPU2→GPU5→GPU0→GPU6; Ring640: GPU3→GPU7→GPU4→GPU5→GPU2→GPU0→GPU6; Ring641: GPU3→GPU7→GPU4→GPU5→GPU0→GPU2→GPU6; Ring642: GPU3→GPU4→GPU7→GPU5→GPU0→GPU2→GPU6; Ring643: GPU3→GPU4→GPU5→GPU7→GPU0→GPU2→GPU6; Ring644: GPU3→GPU4→GPU5→GPU0→GPU7→GPU2→GPU6; Ring645: GPU3→GPU4→GPU5→GPU0→GPU2→GPU7→GPU6; Ring646: GPU3→GPU5→GPU4→GPU0→GPU2→GPU7→GPU6; Ring647: GPU3→GPU5→GPU0→GPU4→GPU2→GPU7→GPU6; Ring648: GPU3→GPU5→GPU0→GPU2→GPU4→GPU7→GPU6.

[0134] In FIG. 4b, only the Ring633 of the above created 6:2 type communication rings is shown in the computing node node3.

[0135] And correspondingly, when creating 16 6:2 type communication rings in the computing node node4, GPU6 and GPU3 can be selected as the start GPU and the end GPU from the 8 GUPs contained in the computing node node3, and then five GPUs are randomly selected from the remaining GPU0, GPU1, GPU2, GPU3, GPU4 and GPU7 as intermediate GPUs, for example, GPU7, GPU5, GPU4, GPU1 and GPU0 can be selected as intermediate GPUs. From the above, one form of the 6:2 type communication ring created in the computing node node3 can be: GPU6→……→GPU3, wherein there are five GPUs between GPU6 and GPU3 and the arrangement order of the five GPUs (as intermediate GPUs) is not limited.Exemplarily, the 16 6:2 type communication rings created in the computing node node3 can be, but are not limited to, Ring649~Ring664 as shown below: Ring649: GPU6→GPU7→GPU5→GPU4→GPU1→GPU0→GPU3; Ring650: GPU6→GPU5→GPU7→GPU4→GPU1→GPU0→GPU3; Ring651: GPU6→GPU5→GPU4→GPU7→GPU1→GPU0→GPU3; Ring652: GPU6→GPU5→GPU4→GPU1→GPU7→GPU0→GPU3; Ring653: GPU6→GPU5→GPU4→GPU1→GPU0→GPU7→GPU3; Ring654: GPU6→GPU4→GPU5→GPU1→GPU0→GPU7→GPU3; Ring655: GPU6→GPU4→GPU1→GPU5→GPU0→GPU7→GPU3; Ring656: GPU6→GPU4→GPU1→GPU0→GPU5→GPU7→GPU3; Ring657: GPU6→GPU4→GPU1→GPU0→GPU7→GPU5→GPU3; Ring658: GPU6→GPU1→GPU4→GPU0→GPU7→GPU5→GPU3; Ring659: GPU6→GPU1→GPU0→GPU4→GPU7→GPU5→GPU3; Ring660: GPU6→GPU1→GPU0→GPU7→GPU4→GPU5→GPU3; Ring661: GPU6→GPU1→GPU0→GPU7→GPU5→GPU4→GPU3; Ring662: GPU6→GPU0→GPU1→GPU7→GPU5→GPU4→GPU3; Ring663: GPU6→GPU0→GPU7→GPU1→GPU5→GPU4→GPU3; Ring664: GPU6→GPU0→GPU7→GPU5→GPU1→GPU4→GPU3.

[0136] In FIG. 4b, only the 6:2 type communication ring Ring649 created above is shown in the computing node node4.

[0137] 3) When 32 4:4 type communication rings are created in the worker node, based on the ring type information of this type of communication ring such as the ratio 4:4, it can be known that 5 GPUs in the worker node need to be used.

[0138] (a13) In the network comprising four worker nodes of compute node nodeO, compute node nodel, compute node node2 and compute node node3, as shown in FIG. 5a, when 32 4:4 type communication rings are created in compute node nodeO, exemplary, GPUO and GPU4 can be first randomly selected from the eight GUPs comprised in compute node nodeO as the start GPU and the end GPU, respectively, and then three GPUs are randomly selected from the remaining GPUs of GPUl, GPU2, GPU3, GPU5, GPU6 and GPU7 as intermediate GPUs, i.e., one form of the 4:4 type communication ring created in compute node nodeO can be: GPUO→.......→GPU4, wherein there are three different GPUs between GPUO and GPU4 and the arrangement order of the three GPUs is not limited.

[0139] Since the arrangement order of the three different GPUs is at most six, in order to construct 32 4:4 type communication rings, a plurality of groups of different GPUs can be selected as intermediate GPUs, each group of GUPs comprising three GPUs, and at least part of the GPUs included in different groups of GPUs are different.

[0140] For example, based on the above example, on the basis of maintaining GPUO and GPU4 as the start GPU and the end GPU, respectively, a plurality of groups of different GPUs can be randomly selected from the remaining GPUs of GPUl, GPU2, GPU3, GPU5, GPU6 and GPU7 as intermediate GPUs, such as, but not limited to, the following six groups of GPUs can be selected as intermediate GPUs.

[0141] The first group of GPUs: GPU3, GPU2 and GPUl; the second group of GPUs: GPU5, GPU2 and GPUl.

[0142] The third group of GPUs: GPU6, GPU2 and GPUl; the fourth group of GPUs: GPU7, GPU2 and GPUl.

[0143] The fifth group of GPUs: GPU5, GPU6 and GPU7; the sixth group of GPUs: GPUl, GPU6 and GPU7.

[0144] Correspondingly, using the first group of GPUs as intermediate GPUs in the computing node node0, the following 6 4:4 type communication rings in the form of GPU0→.......→GPU4 can be constructed: Ring41 : GPU0→GPU3→GPU2→GPU1→GPU4; Ring42: GPU0→GPU2→GPU3→GPU1→GPU4; Ring43: GPU0→GPU2→GPU1→GPU3→GPU4; Ring44: GPU0→GPU1→GPU2→GPU3→GPU4; Ring45: GPU0→GPU1→GPU3→GPU2→GPU4; Ring46: GPU0→GPU3→GPU1→GPU2→GPU4.

[0145] Using the second group of GPUs as intermediate GPUs in the computing node node0, the following 6 4:4 type communication rings in the form of GPU0→.......→GPU4 can be constructed: Ring47: GPU0→GPU5→GPU2→GPU1→GPU4; Ring48: GPU0→GPU2→GPU5→GPU1→GPU4; Ring49: GPU0→GPU2→GPU1→GPU5→GPU4; Ring410: GPU0→GPU1→GPU2→GPU5→GPU4; Ring411: GPU0→GPU1→GPU5→GPU2→GPU4; Ring412: GPU0→GPU5→GPU1→GPU2→GPU4.

[0146] Using the third group of GPUs as intermediate GPUs in the computing node node0, the following 6 4:4 type communication rings in the form of GPU0→.......→GPU4 can be constructed: Ring413: GPU0→GPU6→GPU2→GPU1→GPU4; Ring414: GPU0→GPU2→GPU6→GPU1→GPU4; Ring415: GPU0→GPU2→GPU1→GPU6→GPU4; Ring416: GPU0→GPU1→GPU2→GPU6→GPU4; Ring417: GPU0→GPU1→GPU6→GPU2→GPU4; Ring418: GPU0→GPU6→GPU1→GPU2→GPU4.

[0147] Using the above fourth group of GPUs as intermediate GPUs within the compute node nodeO, the following 6 4:4 type communication rings in the form of GPU0→.......→GPU4 can be constructed: Ring419: GPU0→GPU7→GPU2→GPU1→GPU4; Ring420: GPU0→GPU2→GPU7→GPU1→GPU4; Ring421: GPU0→GPU2→GPU1→GPU7→GPU4; Ring422: GPU0→GPU1→GPU2→GPU7→GPU4; Ring423: GPU0→GPU1→GPU7→GPU2→GPU4; Ring424: GPU0→GPU7→GPU1→GPU2→GPU4.

[0148] Using the above fifth group of GPUs as intermediate GPUs within the compute node nodeO, the following 6 4:4 type communication rings in the form of GPU0→.......→GPU4 can be constructed: Ring425: GPU0→GPU5→GPU6→GPU7→GPU4; Ring426: GPU0→GPU6→GPU5→GPU7→GPU4; Ring427: GPU0→GPU6→GPU7→GPU5→GPU4; Ring428: GPU0→GPU7→GPU6→GPU5→GPU4; Ring429: GPU0→GPU7→GPU5→GPU6→GPU4; Ring430: GPU0→GPU5→GPU7→GPU6→GPU4.

[0149] Using the above sixth group of GPUs as intermediate GPUs within the compute node nodeO, the following 2 4:4 type communication rings in the form of GPU0→.......→GPU4 can be constructed: Ring431: GPU0→GPU1→GPU6→GPU7→GPU4; Ring432: GPU0→GPU6→GPU1→GPU7→GPU4.

[0150] Using the above sixth group of GPUs, other 4:4 type communication rings in the form of GPU0→.......→GPU4 can also be constructed, such as, GPU0→GPU6→GPU7→GPU1→GPU4 and other 4:4 type communication rings.

[0151] From the above, the 32 4:4 type communication rings constructed within the compute node nodeO can all be in the form of GPU0→.......→GPU4, such as the above exemplary given Ring41~Ring432. In FIG. 5a, only one of the above created 4:4 type communication rings, Ring41, is shown within the compute node nodeO.

[0152] Accordingly, in the computing node node1, GPU4 and GPU0 can be randomly selected from the 8 GUPs contained in the computing node node0 as the start GPU and the end GPU respectively, and then three GPUs can be randomly selected from the remaining six GPUs GPU1, GPU2, GPU3, GPU5, GPU6 and GPU7 as the intermediate GPUs. Thus, one form of the 4:4 type communication ring created in the computing node node1 can be: GPU4→...→GPU0, wherein there are three different GPUs between GPU4 and GPU0, and the arrangement order of the three GPUs is not limited.

[0153] Since the order of the three different GPUs is at most 6, in order to be able to construct 32 4:4 type communication rings, the same way as the computing node node0 can be used to construct 32 4:4 type communication rings in the computing node node1. Specifically, referring to the way given for the computing node node0, for example, based on the above example, on the basis of maintaining the use of GPU4 and GPU0 as the start GPU and the end GPU respectively, a plurality of groups of different GPUs can be randomly selected from the remaining six GPUs GPU1, GPU2, GPU3, GPU5, GPU6 and GPU7 as intermediate GPUs, wherein each group of GPUs contains three different GPUs, and at least part of the GPUs included in different groups of GPUs are different. For example, the following six groups of GPUs can be selected as intermediate GPUs.

[0154] The first group of GPUs: GPU7, GPU2 and GPU3; the second group of GPUs: GPU1, GPU2 and GPU3.

[0155] The third group of GPUs: GPU5, GPU2 and GPU3; the fourth group of GPUs: GPU6, GPU2 and GPU3.

[0156] The fifth group of GPUs: GPU5, GPU6 and GPU1; the sixth group of GPUs: GPU5, GPU6 and GPU7.

[0157] Correspondingly, using the first group of GPUs as intermediate GPUs in the computing node node1, the following 6 4:4 type communication rings in the form of GPU4→.......→GPU0 can be constructed: Ring433: GPU4→GPU7→GPU2→GPU3→GPU0; Ring434: GPU4→GPU2→GPU7→GPU3→GPU0; Ring435: GPU4→GPU2→GPU3→GPU7→GPU0; Ring436: GPU4→GPU3→GPU2→GPU7→GPU0; Ring437: GPU4→GPU3→GPU7→GPU2→GPU0; Ring438: GPU4→GPU7→GPU3→GPU2→GPU0.

[0158] Using the second group of GPUs as intermediate GPUs in the computing node node1, the following 6 4:4 type communication rings in the form of GPU4→.......→GPU0 can be constructed: Ring439: GPU4→GPU1→GPU2→GPU3→GPU0; Ring440: GPU4→GPU2→GPU1→GPU3→GPU0; Ring441: GPU4→GPU2→GPU3→GPU1→GPU0; Ring442: GPU4→GPU3→GPU2→GPU1→GPU0; Ring443: GPU4→GPU3→GPU1→GPU2→GPU4; Ring444: GPU4→GPU1→GPU3→GPU2→GPU0.

[0159] Using the third group of GPUs as intermediate GPUs in the computing node node1, the following 6 4:4 type communication rings in the form of GPU4→.......→GPU0 can be constructed: Ring445: GPU4→GPU5→GPU2→GPU3→GPU0; Ring446: GPU4→GPU2→GPU5→GPU3→GPU0; Ring447: GPU4→GPU2→GPU3→GPU5→GPU0; Ring448: GPU4→GPU3→GPU2→GPU5→GPU0; Ring449: GPU4→GPU3→GPU5→GPU2→GPU4; Ring450: GPU4→GPU5→GPU3→GPU2→GPU0.

[0160] When the fourth group of GPUs is used as intermediate GPUs at the computing node node1, six 4:4 type communication rings in the form of GPU4→.......→GPU0 can be constructed as follows: Ring451: GPU4→GPU6→GPU2→GPU3→GPU0; Ring452: GPU4→GPU2→GPU6→GPU3→GPU0; Ring453: GPU4→GPU2→GPU3→GPU6→GPU0; Ring454: GPU4→GPU3→GPU2→GPU6→GPU0; Ring455: GPU4→GPU3→GPU6→GPU2→GPU4; Ring456: GPU4→GPU6→GPU3→GPU2→GPU0.

[0161] When the fifth group of GPUs is used as intermediate GPUs at the computing node node1, six 4:4 type communication rings in the form of GPU4→.......→GPU0 can be constructed as follows: Ring457: GPU4→GPU5→GPU6→GPU1→GPU0; Ring458: GPU4→GPU6→GPU5→GPU1→GPU0; Ring459: GPU4→GPU6→GPU1→GPU5→GPU0; Ring460: GPU4→GPU1→GPU6→GPU5→GPU0; Ring461: GPU4→GPU1→GPU5→GPU6→GPU4; Ring462: GPU4→GPU5→GPU1→GPU6→GPU0.

[0162] When the sixth group of GPUs is used as intermediate GPUs at the computing node node1, two 4:4 type communication rings in the form of GPU4→.......→GPU0 can be constructed as follows: Ring463: GPU4→GPU5→GPU6→GPU7→GPU0; Ring464: GPU4→GPU6→GPU5→GPU7→GPU0.

[0163] It should be noted that the sixth group of GPUs can also be used to construct other 4:4 type communication rings in the form of GPU4→.......→GPU0, such as a GPU4→GPU6→GPU7→GPU5→GPU4 type communication ring.

[0164] As shown above, the 32 4:4 type communication rings constructed in the computing node node1 are all in the form of GPU4→.......→GPU0, such as the Ring433 to Ring464 shown in the above examples. In FIG. 5a, only the Ring433 constructed above is shown in the computing node node1.

[0165] For creating 32 4:4 type communication rings in the computing node node2, refer to the foregoing content related to creating 32 4:4 type communication rings in the computing node node0, and for creating 32 4:4 type communication rings in the computing node node3, refer to the foregoing content related to creating 32 4:4 type communication rings in the computing node node1.

[0166] (b13) In the network containing five working nodes of the computing node node0, the computing node node1, the computing node node2, the computing node node3 and the computing node node4, the implementation of creating 32 4:4 type communication rings in the computing node node0, the computing node node1 and the computing node node2 can refer to the foregoing content given in (a13) and will not be described here. For example, the 32 4:4 type communication rings created in the computing node node0 and the computing node node2 can all be in the form of GPU0→.......→GPU4, such as Ring41 to Ring432 given in (a13), and the 32 4:4 type communication rings created in the computing node node1 can all be in the form of GPU4→.......→GPU0, such as Ring433 to Ring464 given in (a13).

[0167] Based on the foregoing examples, when 32 4:4 type communication rings are created in the computing node node3, GPU4 is selected as the starting GPU from the 8 GPUs contained in the computing node node3, and one GPU is selected as the terminal GPU from the GPUs other than GPU4 and GPU0, such as GPU1, and then multiple groups of GPUs are selected from GPU0, GPU2, GPU3, GPU5, GPU6 and GPU7 as intermediate GPUs to create 32 4:4 type communication rings in the form of GPU4→.......→GPU1, wherein each group of GPUs includes three different GPUs, and at least part of the GPUs in different groups are different. For example, the multiple groups of GPUs can be but not limited to the following groups of GPUs.

[0168] The first group of GPUs: GPU6, GPU7, GPU2; the second group of GPUs: GPU0, GPU7, GPU2.

[0169] The third group of GPUs: GPU3, GPU7, GPU2; the fourth group of GPUs: GPU5, GPU7, GPU2.

[0170] Fifth group of GPUs: GPU0, GPU3, GPU5; sixth group of GPUs: GPU2, GPU3, GPU5.

[0171] wherein, using the above first group of GPUs in combination with GPU4, GPU1 within the compute node node3, the following 6 GPU4→.......→GPU1 form 4:4 type communication rings can be constructed: GPU4→GPU6→GPU7→GPU2→GPU1; GPU4→GPU7→GPU6→GPU2→GPU1; GPU4→GPU7→GPU2→GPU6→GPU1; GPU4→GPU2→GPU7→GPU6→GPU1; GPU4→GPU2→GPU6→GPU7→GPU1; GPU4→GPU6→GPU2→GPU7→GPU1.

[0172] wherein, using the above second group of GPUs in combination with GPU4, GPU1 within the compute node node3, the following 6 GPU4→.......→GPU1 form 4:4 type communication rings can be constructed: GPU4→GPU0→GPU7→GPU2→GPU1; GPU4→GPU7→GPU0→GPU2→GPU1; GPU4→GPU7→GPU2→GPU0→GPU1; GPU4→GPU2→GPU7→GPU0→GPU1; GPU4→GPU2→GPU0→GPU7→GPU1; GPU4→GPU0→GPU2→GPU7→GPU1.

[0173] wherein, using the above third group of GPUs in combination with GPU4, GPU1 within the compute node node3, the following 6 GPU4→.......→GPU1 form 4:4 type communication rings can be constructed: GPU4→GPU3→GPU7→GPU2→GPU1; GPU4→GPU7→GPU3→GPU2→GPU1; GPU4→GPU7→GPU2→GPU3→GPU1; GPU4→GPU2→GPU7→GPU3→GPU1; GPU4→GPU2→GPU3→GPU7→GPU1; GPU4→GPU3→GPU2→GPU7→GPU1.

[0174] Using the above fourth group of GPUs in combination with GPU4 and GPU1 within the compute node node3, the following 6 4:4 type communication rings in the form of GPU4→.......→GPU1 can be constructed: GPU4→GPU5→GPU7→GPU2→GPU1; GPU4→GPU7→GPU5→GPU2→GPU1; GPU4→GPU7→GPU2→GPU5→GPU1; GPU4→GPU2→GPU7→GPU5→GPU1; GPU4→GPU2→GPU5→GPU7→GPU1; GPU4→GPU5→GPU2→GPU7→GPU1.

[0175] Using the above fifth group of GPUs in combination with GPU4 and GPU1 within the compute node node3, the following 6 4:4 type communication rings in the form of GPU4→.......→GPU1 can be constructed: GPU4→GPU0→GPU3→GPU5→GPU1; GPU4→GPU3→GPU0→GPU5→GPU1; GPU4→GPU3→GPU5→GPU0→GPU1; GPU4→GPU5→GPU3→GPU0→GPU1; GPU4→GPU5→GPU0→GPU3→GPU1; GPU4→GPU0→GPU5→GPU3→GPU1.

[0176] Using the above sixth group of GPUs in combination with GPU4 and GPU1 within the compute node node3, the following 2 4:4 type communication rings in the form of GPU4→.......→GPU1 can be constructed: GPU4→GPU2→GPU3→GPU5→GPU1; GPU4→GPU3→GPU2→GPU5→GPU1.

[0177] It should be noted here that the above sixth group of GPUs can also be used to construct other 4:4 type communication rings in the form of GPU4→.......→GPU1, such as GPU4→GPU3→GPU5→GPU2→GPU1 and other 4:4 type communication rings.

[0178] In FIG. 5b, only the above created 4:4 type communication ring GPU4→GPU6→GPU7→GPU2→GPU1 is shown within the compute node node3.

[0179] When the 32 4:4 type communication rings are created in the computing node node4, GPU1 and GPU4 are selected as the start GPU and the end GPU respectively from the 8 GPUs contained in the computing node node4, and then multiple groups of GPUs are selected from GPU0, GPU2, GPU3, GPU5, GPU6 and GPU7 as intermediate GPUs to combine with GPU1 and GPU4 to create 32 4:4 type communication rings in the form of GPU1→GPU4; each group of GPUs contains 3 different GPUs, and at least part of the GPUs in different groups are different. For example, the selected multiple groups of GPUs can be but not limited to the following groups of GPUs.

[0180] The first group of GPUs: GPU3, GPU6, GPU7; the second group of GPUs: GP0, GPU6, GPU7.

[0181] The third group of GPUs: GPU2, GPU6, GPU7; the fourth group of GPUs: GP5, GPU2, GPU3.

[0182] The fifth group of GPUs: GPU0, GPU2, GPU3; the sixth group of GPUs: GP7, GPU2, GPU3.

[0183] In which, using the above first group of GPUs in the computing node node4 in combination with GPU1 and GPU4 can construct the following 6 4:4 type communication rings in the form of GPU1→GPU4: GPU1→GPU3→GPU6→GPU7→GPU4; GPU1→GPU6→GPU3→GPU7→GPU4; GPU1→GPU6→GPU7→GPU3→GPU4; GPU1→GPU7→GPU6→GPU3→GPU4; GPU1→GPU7→GPU3→GPU6→GPU4; GPU1→GPU3→GPU7→GPU6→GPU4.

[0184] Using the above second group of GPUs in the computing node node4 in combination with GPU4 and GPU1 can construct the following 6 4:4 type communication rings in the form of GPU1→GPU4: GPU1→GPU0→GPU6→GPU7→GPU4; GPU1→GPU6→GPU0→GPU7→GPU4; GPU1→GPU6→GPU7→GPU0→GPU4; GPU1→GPU7→GPU6→GPU0→GPU4; GPU1→GPU7→GPU0→GPU6→GPU4; GPU1→GPU0→GPU7→GPU6→GPU4.

[0185] Using the third set of GPUs described above in combination with GPU4, GPU1 within the compute node node4, the following 6 GPU1→.......→GPU4 form 4:4 type communication rings can be constructed: GPU1→GPU2→GPU6→GPU7→GPU4; GPU1→GPU6→GPU2→GPU7→GPU4; GPU1→GPU6→GPU7→GPU2→GPU4; GPU1→GPU7→GPU6→GPU2→GPU4; GPU1→GPU7→GPU2→GPU6→GPU4; GPU1→GPU2→GPU7→GPU6→GPU4.

[0186] Using the fourth set of GPUs described above in combination with GPU4, GPU1 within the compute node node4, the following 6 GPU1→.......→GPU4 form 4:4 type communication rings can be constructed: GPU1→GPU5→GPU2→GPU3→GPU4; GPU1→GPU2→GPU5→GPU3→GPU4; GPU1→GPU2→GPU3→GPU5→GPU4; GPU1→GPU3→GPU2→GPU5→GPU4; GPU1→GPU3→GPU5→GPU2→GPU4; GPU1→GPU5→GPU3→GPU2→GPU4.

[0187] Using the fifth set of GPUs described above in combination with GPU4, GPU1 within the compute node node4, the following 6 GPU1→.......→GPU4 form 4:4 type communication rings can be constructed: GPU1→GPU0→GPU2→GPU3→GPU4; GPU1→GPU2→GPU0→GPU3→GPU4; GPU1→GPU2→GPU3→GPU0→GPU4; GPU1→GPU3→GPU2→GPU0→GPU4; GPU1→GPU3→GPU0→GPU2→GPU4; GPU1→GPU0→GPU3→GPU2→GPU4.

[0188] Using the sixth set of GPUs described above in combination with GPU4, GPU1 within the compute node node4, the following 6 GPU1→.......→GPU4 form 4:4 type communication rings can be constructed: GPU1→GPU7→GPU2→GPU3→GPU4; GPU1→GPU2→GPU7→GPU3→GPU4; GPU1→GPU2→GPU3→GPU7→GPU4; GPU1→GPU3→GPU2→GPU7→GPU4; GPU1→GPU3→GPU7→GPU2→GPU4; GPU1→GPU7→GPU3→GPU2→GPU4.

[0189] Further, the method provided by the embodiment further includes: connecting the node-level communication rings among the plurality of worker nodes based on the connection strategies of the communication rings of various types to construct the cross-node communication rings of the network.

[0190] The cross-node communication ring is formed by selecting one node-level communication ring of the same type from the plurality of worker nodes and connecting the selected node-level communication rings in sequence.

[0191] The total number of the cross-node communication rings of the network constructed finally is related to the total number of the communication rings to be created in the worker nodes, and the two are equal.

[0192] FIGS. 3a, 4a and 5a show an example of a cross-node communication ring constructed when the total number of worker nodes in the network is even; and FIGS. 3b, 4b and 5b show an example of a cross-node communication ring constructed when the total number of worker nodes in the network is odd.

[0193] In the embodiment, the number of the communication rings of various types to be created in the worker nodes is an integral multiple of the number of network cards in the worker nodes, so that the bandwidth of the network cards can be evenly allocated when the network cards are allocated to the communication rings, thereby evenly allocating the bandwidth in the worker nodes.

[0194] That is, the first type of communication ring is one of the plurality of types of communication rings to be created in the worker nodes, and the number of the first type of communication rings is an integral multiple of the number of network cards in the worker nodes. The method provided by the embodiment further includes the following step S31.

[0195] S31, allocating the network cards in the first worker node to the node-level communication rings of the first type of communication ring in the first worker node.

[0196] The first worker node is one of the plurality of worker nodes and includes a plurality of network cards, and each network card is allocated to the same number of the first type of communication ring.

[0197] In specific implementation, the step S31 can be performed according to the correspondence information of the node-level communication rings and the network cards input by the user through the client device, or can be performed according to the preset allocation strategy (such as the uniform allocation strategy) for the node-level communication rings and the network cards, which is not limited in the embodiment.

[0198] For example, if there are 8 network cards in each worker node and 8 first type communication rings are constructed in each worker node, the 8 first type communication rings in the first worker node can be assigned to the network cards in the first worker node in a one-to-one relationship, i.e., one network card is assigned to one first type communication ring.

[0199] FIG. 7 shows a flowchart of a communication ring generation method according to another embodiment of the present disclosure. As shown in FIG. 7, the communication ring generation method includes the following steps.

[0200] 201, obtaining worker node information of a network; wherein the network includes a plurality of worker nodes, and a plurality of node-level communication rings are generated in the plurality of worker nodes.

[0201] 202, determining a communication ring connection strategy between the plurality of worker nodes according to the worker node information.

[0202] 203, connecting a plurality of node-level communication rings between the plurality of worker nodes based on the communication ring connection strategy to construct a plurality of cross-node communication rings of the network.

[0203] In the above embodiment, the plurality of node-level communication rings include a plurality of types of communication rings, and the number of each type of communication ring is determined according to bandwidth occupation information corresponding to each type of communication ring, worker node information of the network, and a set communication ring bandwidth, wherein the bandwidth occupation information corresponding to each type of communication ring includes node-in-bandwidth occupation and inter-node network bandwidth occupation when data is transmitted through the type of communication ring.

[0204] Further, the plurality of node-level communication rings comprises a plurality of types of communication rings, and the quantity information of the plurality of node-level communication rings comprises quantity information of the plurality of types of communication rings; and the worker node information comprises a total quantity of worker nodes and node numbers of the worker nodes. The determined communication ring connection strategy comprises connection strategies of the plurality of types of communication rings among the plurality of worker nodes. A first type of communication ring is one of the plurality of types of communication rings. Step 202 “determining a connection strategy of a first type of communication ring among the plurality of worker nodes according to worker node information of the network” comprises step 2021.

[0205] 2021. According to the parity of the total quantity of worker nodes and the parity of the node numbers of the plurality of worker nodes, determine a first type of communication ring to be used by each of the plurality of worker nodes when constructing a cross-node communication ring, to connect the first type of communication ring at the beginning and the end.

[0206] For specific implementation of each step described above in this embodiment, refer to related content in other embodiments. In addition to the steps described above, the method provided by the embodiments of the present disclosure can further comprise other steps. For other steps that can be further included and specific implementation, refer to related content in other embodiments, which will not be repeated here.

[0207] Using the plurality of cross-node communication rings constructed in the above other embodiments, parallel data parallel processing can be performed, for example, training sample data can be processed in parallel when training a model. Therefore, the present disclosure further provides a model training method. The model training method comprises the following steps.

[0208] 301. Obtain a training sample data set.

[0209] 302. Divide the training sample data set into a total quantity of data blocks according to the total quantity of the plurality of cross-node communication rings in the network.

[0210] 303. Assign each cross-node communication ring in the plurality of cross-node communication rings a data block, so that each cross-node communication ring trains a target model using the assigned data block.

[0211] 304. According to the training result of each cross-node communication ring on the target model, optimize the target model.

[0212] The network comprises a plurality of worker nodes, a plurality of node-level communication rings are generated in each worker node, and the target model is deployed in each worker node. The plurality of node-level communication rings comprise different types of communication rings. One node-level communication ring is formed by connecting at least part of the processors in the plurality of processors in the corresponding worker node in a ring. One cross-node communication ring is formed by selecting one node-level communication ring belonging to the same type of communication ring from the plurality of worker nodes and sequentially connecting the selected node-level communication rings end to end. The total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings.

[0213] In the above 301-302, each training sample data in the training sample data set comprises a training sample and a true label of the training sample. The training sample data can be obtained from a network database, but is not limited thereto. For details of the construction of the plurality of cross-node communication rings, please refer to the related contents in other embodiments. According to the total number of the plurality of cross-node communication rings, the training sample data set can be divided into the same number of data blocks, but is not limited thereto.

[0214] In the above 303-304, each cross-node communication ring uses the target model to perform forward propagation calculation, loss calculation, reverse transmission calculation and other operations when processing the assigned data block, so as to finally obtain the corresponding training result. The training result can comprise gradient information of each parameter of the target model. When optimizing the target model, the training results (gradient information of the parameters) of each cross-node communication ring on the target model can be averaged, and then the parameters of the target model are updated using the averaged training result, so as to realize the optimization of the parameters of the target model.

[0215] For details of the specific implementation of the cross-node communication ring, the plurality of node-level communication rings and the like involved in the present embodiment, please refer to the related contents in other embodiments.

[0216] The network system further comprises: a plurality of worker nodes; each of the worker nodes comprises a plurality of processors; a control node configured to: obtain ring type information of a plurality of types of communication rings to be created in each of the worker nodes, wherein the ring type information of a type of communication ring is capable of reflecting a number of processors to be used when creating the type of communication ring and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information comprises a first value representing bandwidth occupation in a node and a second value representing network bandwidth occupation between nodes; determine constraint information and an objective function according to the bandwidth occupation information of the plurality of types of communication rings, worker node information of the network, and a set communication ring bandwidth and a parameter representing a number of the plurality of types of communication rings to be generated in each of the worker nodes; solve the objective function based on the constraint information to obtain the number of the plurality of types of communication rings to be generated in each of the worker nodes; and perform, for each of the plurality of worker nodes, a ring connection operation of selecting an adaptive number of processors from a plurality of processors in the worker node based on the number of processors to be used when creating the plurality of types of communication rings and the number of the plurality of types of communication rings to be generated in each of the worker nodes, to generate a plurality of node-level communication rings in the worker node; and connect the plurality of node-level communication rings between the plurality of worker nodes to construct a plurality of cross-node communication rings of the network, wherein a total number of the plurality of cross-node communication rings is equal to a total number of the plurality of node-level communication rings, and each of the cross-node communication rings is formed by selecting one node-level communication ring of a same type from the plurality of worker nodes and sequentially connecting the selected node-level communication rings.

[0217] The network system can be used for distributed model training, and the plurality of processors in each of the worker nodes can be GPUs. In addition to the steps described above, the control node can further perform steps related to other embodiments of the present disclosure, such as the steps in the communication ring generation method shown in FIG. 7 and the steps in the model training method described in other embodiments.

[0218] For details of the specific forms of the worker nodes and the control node and the specific implementation of the functions of the control node, please refer to the related content in other embodiments of the present disclosure, which will not be described here.

[0219] The present disclosure further provides a communication ring generation device, which can be deployed in the control node of the network system described above. FIG. 8 shows a structural schematic diagram of the communication ring generation device. As shown in FIG. 8, the communication ring generation device comprises an obtaining module 31, a determining module 32, a generating module 33, and a connecting module 34.

[0220] The acquisition module 31 is configured to acquire ring type information of multiple types of communication rings to be created in each working node in a network; wherein the ring type information of one type of communication ring can reflect a number of processors to be used when creating the type of communication ring, and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information including a first value representing occupied bandwidth in a node and a second value representing occupied bandwidth between nodes in the network; the network includes multiple working nodes.

[0221] The determination module 32 is configured to determine the number of various types of communication rings to be generated in each working node according to the ring type information of various types of communication rings, working node information of the network, and a set communication ring bandwidth and a parameter representing the number of various types of communication rings to be generated in each working node.

[0222] The generation module 33 is configured to perform a ring connection operation of selecting an adaptive number of processors from multiple processors in a working node based on the number of processors to be used when creating the various types of communication rings and the number of various types of communication rings to be generated in each working node, to generate multiple node-level communication rings in the multiple working nodes, respectively.

[0223] The connection module 34 is configured to connect multiple node-level communication rings between the multiple working nodes to construct multiple cross-node communication rings of the network; wherein the total number of the multiple cross-node communication rings is equal to the total number of the multiple node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring of the same type from the multiple working nodes and sequentially connecting the selected node-level communication rings.

[0224] Further, the determination module 32 is configured to determine the number of various types of communication rings to be generated in each working node according to the ring type information of various types of communication rings, working node information of the network, and a set communication ring bandwidth and a parameter representing the number of various types of communication rings to be generated in each working node; specifically, the determination module 32 is configured to determine constraint information according to the bandwidth occupation information corresponding to the various types of communication rings, the working node information, and the parameter representing the number of various types of communication rings to be generated in each working node; determine a target function according to the parameter representing the number of various types of communication rings to be generated in each working node and the communication ring bandwidth; and solve the target function based on the constraint information to obtain the number of various types of communication rings to be generated in each working node.

[0225] Further, the worker node information includes an intra-node total bandwidth and an intra-node NIC number of a single worker node, an inter-node total network bandwidth of two worker nodes; the objective function is to maximize a product value between a target sum value and the communication ring bandwidth, the target sum value is equal to a sum value between parameters representing a number of various types of communication rings that need to be generated in each worker node; the constraint information includes an intra-node bandwidth constraint, an inter-node network bandwidth constraint, and a bandwidth uniform distribution constraint; the intra-node bandwidth constraint is established based on the parameters representing the number of various types of communication rings that need to be generated in each worker node, the first numerical value corresponding to various types of communication rings, the communication ring bandwidth, and the intra-node total bandwidth; the inter-node network bandwidth constraint is established based on the parameters representing the number of various types of communication rings that need to be generated in each worker node, the second numerical value corresponding to various types of communication rings, the communication ring bandwidth, and the inter-node total network bandwidth; and the bandwidth uniform distribution constraint is established based on the parameters representing the number of various types of communication rings that need to be generated in each worker node, the second numerical value corresponding to various types of communication rings, and the intra-node NIC number.

[0226] Further, the above-mentioned generation module 33, when used for respectively executing, for the plurality of worker nodes, a ring connection operation of selecting an adaptive number of processors from a plurality of processors in a worker node based on the number of processors that need to be used at the time of creation and the number of various types of communication rings that need to be generated in each worker node, to respectively generate a plurality of node-level communication rings in the plurality of worker nodes, is specifically used for: determining a connection strategy of various types of communication rings between the plurality of worker nodes according to the worker node information of the network; and based on the connection strategy of various types of communication rings, the number of processors that need to be used at the time of creation, and the number of various types of communication rings that need to be generated in each worker node, respectively executing, for the plurality of worker nodes, a ring connection operation of selecting an adaptive number of processors from a plurality of processors in a worker node, to respectively generate a plurality of node-level communication rings in the plurality of worker nodes.

[0227] Further, the above-mentioned connection module 34 is also used for triggering the connection of a plurality of node-level communication rings between a plurality of worker nodes to construct a plurality of cross-node communication rings of the network based on the connection strategy of various types of communication rings. Wherein, one of the cross-node communication rings is formed by selecting one node-level communication ring of the same type from the plurality of worker nodes and connecting the selected node-level communication rings end to end.

[0228] Further, the worker node information of the network further comprises a total number of worker nodes, node numbers of each worker node; the first type of communication ring is one of the multiple types of communication rings; and the generation module 33 is specifically configured to: determine, according to the parity of the total number of worker nodes and the parity of the node numbers of the multiple worker nodes, a first type of communication ring to be used by each of the multiple worker nodes when constructing a cross-node communication ring, and connect the first type of communication ring used by each of the multiple worker nodes at the head and tail, so as to determine the connection strategy of the first type of communication ring among the multiple worker nodes according to the worker node information of the network; and at least two processors connected between two of the multiple worker nodes have the same number when connected at the head and tail.

[0229] Further, in the cross-node communication ring, the multiple worker nodes are arranged in a ring shape according to the increasing node numbers; and the generation module 33 is specifically configured to: if the total number of worker nodes N = 2K, K is a positive integer greater than or equal to 1, then all odd-numbered node numbers of the multiple worker nodes need to use a first communication ring and all even-numbered node numbers of the multiple worker nodes need to use a second communication ring when constructing a cross-node communication ring; the first communication ring and the second communication ring both belong to the first type of communication ring, and the terminal processor of the first communication ring and the starting processor of the second communication ring have the same number, and the starting processor of the first communication ring and the terminal processor of the second communication ring have the same number; if the total number of worker nodes is N = 2K + 1, and zero is the starting node number of the worker nodes, then all odd-numbered node numbers of the multiple worker nodes with node numbers less than or equal to 2K-2 need to use a first communication ring, all even-numbered node numbers of the multiple worker nodes need to use a second communication ring, the worker node with node number equal to 2K-1 needs to use a third communication ring, and the worker node with node number equal to 2K needs to use a fourth communication ring when constructing a cross-node communication ring; the starting processor of the third communication ring and the terminal processor of the second communication ring, and the terminal processor of the fourth communication ring have the same number, the number of the terminal processor of the third communication ring is different from the numbers of the starting processor and the terminal processor of the second communication ring, but is the same as the number of the starting processor of the fourth communication ring; the third communication ring and the fourth communication ring both belong to the first type of communication ring.

[0230] Further, the number of the first type of communication rings to be generated in each working node is an integer multiple of the number of network cards in the working node. The apparatus further includes a distribution module configured to distribute the network cards in the first working node to each node-level communication ring belonging to the first type of communication rings in the first working node; wherein the first working node is one of the plurality of working nodes and includes a plurality of network cards, and each network card is distributed to the same number of first type of communication rings.

[0231] For specific implementation of the functions of the modules in the apparatus provided in this embodiment, refer to related content in other embodiments of the present disclosure.

[0232] The present disclosure further provides another communication ring generation apparatus, which can be deployed in a control node in the network system. FIG. 9 shows a structural schematic diagram of the communication ring generation apparatus. As shown in FIG. 9, the communication ring generation apparatus includes an acquisition module 41, a determination module 42 and a connection module 43. The acquisition module 41 is configured to acquire working node information of a network; wherein the network includes a plurality of working nodes, and a plurality of node-level communication rings are generated in the plurality of working nodes; the determination module 42 is configured to determine a communication ring connection strategy between the plurality of working nodes according to the working node information; and the connection module 43 is configured to connect the plurality of node-level communication rings between the plurality of working nodes based on the communication ring connection strategy, to construct a plurality of cross-node communication rings of the network; wherein a node-level communication ring is formed by connecting at least part of processors in a plurality of processors in a corresponding working node in a ring manner; the plurality of node-level communication rings include a plurality of types of communication rings, and the number of each type of communication ring is determined according to bandwidth occupation information corresponding to each type of communication ring, working node information of the network and a set communication ring bandwidth, and a parameter representing the number of each type of communication ring to be generated in each working node; the bandwidth occupation information corresponding to each type of communication ring includes occupied intra-node bandwidth and occupied inter-node network bandwidth when data is transmitted through the type of communication ring; the total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings, and a cross-node communication ring is formed by sequentially connecting the first and last nodes of each node-level communication ring selected from the plurality of working nodes and belonging to the same type of communication ring; and in the cross-node communication ring, two processors connected between at least part of two working nodes in the plurality of working nodes have the same number.

[0233] Further, the plurality of node-level communication rings have a plurality of types of communication rings; the worker node information includes a total number of worker nodes and node numbers of the worker nodes; the determined communication ring connection strategy includes connection strategies of the various types of communication rings among the plurality of worker nodes; the first type of communication ring is one of the plurality of types of communication rings; and the determining module 42, in a process of determining the connection strategy of the first type of communication ring among the plurality of worker nodes according to the worker node information of the network, is specifically configured to: determine, according to the parity of the total number of worker nodes and the parity of the node numbers of the plurality of worker nodes, a first type of communication ring to be used by the plurality of worker nodes when a cross-node communication ring is constructed, to connect the first type of communication ring used by the plurality of worker nodes at the beginning and the end.

[0234] For specific implementation of the functions of the modules in the device provided in this embodiment, refer to related content in other embodiments of this disclosure.

[0235] The present disclosure also provides another model training device, which can be deployed in a control node in the foregoing network system. The model training device can include an obtaining module, a dividing module, an allocating module, and an optimizing module. The obtaining module is configured to obtain a training sample data set. The dividing module is configured to divide the training sample data set into a total number of data blocks according to a total number of a plurality of cross-node communication rings in a network. The allocating module is configured to allocate one of the data blocks to each cross-node communication ring in the plurality of cross-node communication rings, so that each cross-node communication ring trains a target model using the allocated data block. The optimizing module is configured to optimize the target model according to training results of the target model by each cross-node communication ring. The network includes a plurality of worker nodes, each of which generates a plurality of node-level communication rings and deploys the target model. The plurality of node-level communication rings have different types of communication rings. One node-level communication ring is formed by connecting at least part of processors in a corresponding worker node in a ring shape. One cross-node communication ring is formed by sequentially connecting, at the beginning and the end, a plurality of node-level communication rings selected from the plurality of worker nodes and belonging to the same type of communication ring. The total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings.

[0236] For specific implementation of the functions of the modules in the device provided in this embodiment, refer to related content in other embodiments of this disclosure.

[0237] FIG. 10 shows a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 10, the electronic device includes a memory 51 and a processor 52. The memory 51 can be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk. Specifically,

[0238] The memory 51 is configured to store a program; and the processor 52 is coupled to the memory 51 and is configured to execute the program stored in the memory, so as to perform the steps or functions in the methods provided by the embodiments of the present disclosure.

[0239] Further, as shown in FIG. 10, the electronic device further includes a communication component 53, a power supply component 54, an audio component 55, a display (not shown in the figure) and other components. Only part of the components are shown in FIG. 10, which does not mean that the electronic device only includes the components shown in FIG. 10. The electronic device can be a control node in the network system.

[0240] The embodiments of the present disclosure further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a computer, and can implement the method steps provided by the above embodiments.

[0241] The embodiments of the present disclosure further provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor can implement the method steps or functions provided by the above embodiments.

[0242] Those skilled in the art can clearly understand the implementation of the embodiments by means of software and necessary universal hardware platforms through the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the embodiments or some parts of the embodiments.

[0243] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present disclosure, but not to limit them; although the present disclosure has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method of generating a communication ring, wherein, The method comprises: obtaining ring type information of multiple types of communication rings that need to be created in each working node in a network; wherein the ring type information of one type of communication ring can reflect the number of processors that need to be used when creating the type of communication ring, and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information comprising a first value representing the occupation of the bandwidth in a node and a second value representing the occupation of the network bandwidth between nodes; the network comprises multiple working nodes; determining the number of various types of communication rings that need to be generated in each working node according to the bandwidth occupation information corresponding to various types of communication rings, the working node information of the network, and a set communication ring bandwidth and a parameter representing the number of various types of communication rings that need to be generated in each working node; based on the number of processors that need to be used when creating various types of communication rings and the number of various types of communication rings that need to be generated in each working node, selecting an adaptive number of processors from multiple processors in each working node for ring connection operation to generate multiple node-level communication rings in the multiple working nodes respectively; connecting multiple node-level communication rings between the multiple working nodes to construct multiple cross-node communication rings of the network; wherein the total number of the multiple cross-node communication rings is equal to the total number of the multiple node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring of the same type from the multiple working nodes and sequentially connecting the selected node-level communication rings end to end.

2. The method of claim 1, wherein, The method for determining the number of various types of communication rings that need to be generated in each working node according to the bandwidth occupation information corresponding to various types of communication rings, the working node information of the network, and a set communication ring bandwidth and a parameter representing the number of various types of communication rings that need to be generated in each working node, comprises: determining constraint information according to the bandwidth occupation information corresponding to various types of communication rings, the working node information, and the parameter representing the number of various types of communication rings that need to be generated in each working node; determining an objective function according to the parameter representing the number of various types of communication rings that need to be generated in each working node and the communication ring bandwidth; solving the objective function based on the constraint information to obtain the number of various types of communication rings that need to be generated in each working node.

3. The method of claim 2, wherein, The working node information comprises the total bandwidth in a node and the number of network cards in a node of a single working node, and the total network bandwidth between two working nodes; The objective function is to maximize the product value between the target sum value and the communication ring bandwidth, and the target sum value is equal to the sum value between the parameters representing the number of various types of communication rings that need to be generated in each working node; The constraint information comprises node internal bandwidth constraint, inter-node network bandwidth constraint, and bandwidth uniform distribution constraint; The node internal bandwidth constraint is established based on the parameter representing the number of various types of communication rings that need to be generated in each working node, the first value corresponding to various types of communication rings, the communication ring bandwidth, and the total bandwidth in the node. The inter-node network bandwidth constraint is established based on a parameter representing the number of various types of communication rings that need to be generated in each worker node, the second numerical value corresponding to the various types of communication rings, the communication ring bandwidth, and the total inter-node network bandwidth; The bandwidth uniform distribution constraint is established based on a parameter representing the number of various types of communication rings that need to be generated in each worker node, the second numerical value corresponding to the various types of communication rings, and the number of network cards in the worker node.

4. The method of any one of claims 1 to 3, wherein, Based on the number of processors that need to be used when created and the number of various types of communication rings that need to be generated in each worker node, the method comprises: According to the network node information, the connection strategy of various types of communication rings between the plurality of worker nodes is determined; Based on the connection strategy of various types of communication rings, the number of processors that need to be used when created, and the number of various types of communication rings that need to be generated in each worker node, the method comprises: Furthermore, the method comprises: Based on the connection strategy of various types of communication rings, the method triggers the connection of the plurality of node-level communication rings between the plurality of worker nodes to construct a plurality of cross-node communication rings of the network.

5. The method of claim 4, wherein, The network node information further comprises the total number of worker nodes and the node number of each worker node. The first type of communication ring is one of the plurality of types of communication rings; based on the network node information, the connection strategy of the first type of communication ring between the plurality of worker nodes is determined, which comprises: According to the parity of the total number of worker nodes and the parity of the node number of each of the plurality of worker nodes, a first type of communication ring that each of the plurality of worker nodes should use when constructing a cross-node communication ring is determined, so as to connect the first type of communication ring used by the plurality of worker nodes at the beginning and the end. When connected at the beginning and the end, the two processors connected between at least part of two worker nodes in the plurality of worker nodes have the same number.

6. The method of claim 5, wherein, In the cross-node communication ring, the plurality of worker nodes are arranged in a ring according to the increasing node number; and According to the parity of the total number of worker nodes and the parity of the node number of each of the plurality of worker nodes, a first type of communication ring that each of the plurality of worker nodes should use when constructing a cross-node communication ring is determined, which comprises: If the total number of the working nodes N=2K, K is a positive integer greater than or equal to 1, when a cross-node communication ring is constructed, all odd-numbered working nodes in the plurality of working nodes need to use a first communication ring, and all even-numbered working nodes need to use a second communication ring; the first communication ring and the second communication ring both belong to the first type of communication ring, and the terminal processor of the first communication ring and the starting processor of the second communication ring have the same number, and the starting processor of the first communication ring and the terminal processor of the second communication ring have the same number; If the total number of the working nodes N=2K+1, and zero is the starting node number of the working node, when a cross-node communication ring is constructed, all odd-numbered working nodes with a node number less than or equal to 2K-2 in the plurality of working nodes need to use a first communication ring, all even-numbered working nodes need to use a second communication ring, and the working node with a node number equal to 2K-1 needs to use a third communication ring, and the working node with a node number equal to 2K needs to use a fourth communication ring; wherein the starting processor of the third communication ring and the terminal processor of the second communication ring, the terminal processor of the fourth communication ring have the same number, the number of the terminal processor of the third communication ring is different from the numbers of the starting processor and the terminal processor of the second communication ring, but the same as the number of the starting processor of the fourth communication ring; the third communication ring and the fourth communication ring both belong to the first type of communication ring.

7. The method of claim 5, wherein, The number of the first type of communication ring that needs to be generated in each working node is an integer multiple of the number of network cards in the working node; And the method further comprises: allocating the network cards in the first working node to each node-level communication ring belonging to the first type of communication ring in the first working node; Wherein, the first working node is one of the plurality of working nodes and contains a plurality of network cards, and each network card is allocated to the same number of first type communication rings.

8. A method of generating a communication ring, wherein, It includes: Obtain the working node information of the network; wherein the network contains a plurality of working nodes, and a plurality of node-level communication rings are generated in the plurality of working nodes; According to the working node information, determine the communication ring connection strategy between the plurality of working nodes; Based on the communication ring connection strategy, connect the plurality of node-level communication rings between the plurality of working nodes to construct a plurality of cross-node communication rings of the network; Wherein, a node-level communication ring is a ring connection of at least part of the plurality of processors in the corresponding working node; the plurality of node-level communication rings have a plurality of types of communication rings, and the number of each type of communication ring is determined according to the bandwidth occupation information corresponding to each type of communication ring, the working node information of the network, and the set communication ring bandwidth, and the parameter representing the number of each type of communication ring that needs to be generated in each working node; the bandwidth occupation information corresponding to each type of communication ring includes the node internal bandwidth and the inter-node network bandwidth occupied when transmitting data through the type of communication ring. The total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring belonging to the same type of communication ring from the plurality of working nodes respectively and sequentially connecting the selected node-level communication rings in a head-to-tail manner. The working node information includes the total number of working nodes and the node number of each working node.

9. A model training method, wherein, It comprises: Obtaining a training sample data set; According to the total number of the plurality of cross-node communication rings in the network, the training sample data set is divided into a total number of data blocks; Each cross-node communication ring in the plurality of cross-node communication rings is assigned one data block, so that each cross-node communication ring trains the target model using the assigned data block; According to the training result of each cross-node communication ring on the target model, the target model is optimized; The network comprises a plurality of working nodes, a plurality of node-level communication rings are generated in each working node, and the target model is deployed, the plurality of node-level communication rings have different types of communication rings, one node-level communication ring is formed by connecting at least part of the plurality of processors in the corresponding working node in a ring shape, one cross-node communication ring is formed by selecting one node-level communication ring belonging to the same type of communication ring from the plurality of working nodes respectively and sequentially connecting the selected node-level communication rings in a head-to-tail manner, and the total number of the plurality of cross-node communication rings is equal to the total number of the plurality of node-level communication rings.

10. A network system, wherein, It comprises: A plurality of working nodes; The working node comprises a plurality of processors; A control node is configured to obtain ring type information of multiple types of communication rings to be created in each worker node; the ring type information of one type of communication ring can reflect a number of processors to be used when creating the type of communication ring and bandwidth occupation information when transmitting data through the type of communication ring, the bandwidth occupation information including a first value representing occupied bandwidth in a node and a second value representing occupied network bandwidth between nodes; determine a number of the types of communication rings to be generated in each worker node according to the bandwidth occupation information of the types of communication rings, worker node information of a network, and a set communication ring bandwidth and a parameter representing the number of the types of communication rings to be generated in each worker node; based on the number of processors to be used when creating the types of communication rings and the number of the types of communication rings to be generated in each worker node, select an adaptive number of processors from multiple processors in each worker node for ring connection operation to generate multiple node-level communication rings in the multiple worker nodes respectively; and connect the multiple node-level communication rings between the multiple worker nodes to construct multiple cross-node communication rings of the network; the total number of the multiple cross-node-level communication rings is equal to the total number of the multiple node-level communication rings, and one cross-node communication ring is formed by selecting one node-level communication ring of the same type from the multiple worker nodes and sequentially connecting the selected node-level communication rings.

11. An electronic device, comprising: Comprise: a memory and a processor; wherein the memory is configured to store a program; the processor, coupled with the memory, is configured to execute the program stored in the memory to implement the steps in the communication ring generation method of any one of claims 1 to 7 or claim 8, or implement the steps in the model training method of claim 9.

12. A computer readable storage medium, wherein, The computer program is stored in the computer readable storage medium; when the computer program is executed by a computer, the steps in the communication ring generation method of any one of claims 1 to 7 or claim 8 can be implemented, or the steps in the model training method of claim 9 can be implemented.

13. A computer program product comprising a computer program, which, when executed by a processor, causes the processor to implement the steps in the communication ring generation method of any one of claims 1 to 7 or claim 8, or implement the steps in the model training method of claim 9.

Images