Cluster network control method and device

Abstract

The invention discloses a cluster network control method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting the interaction time consumption of information interaction between a target GPU card in each GPU server and target GPU cards in other GPU servers in a process that each GPU server in a cluster network executes each round of large model training task; the target GPU cards in the different GPU servers are used for executing the same subtask of the large model training task; according to the collected interaction time consumption between the target GPU cards, determining the target GPU cards needing interaction optimization; and the transmission priority of the network packet of the target GPU card needing interaction optimization is improved, so that the interaction time consumption of the target GPU card needing interaction optimization is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cluster network control method and device. Background Art

[0002] With the development of artificial intelligence technology, large models are increasingly used. The functions of large models are obtained through training of large models. Large models can be trained using GPU cluster networks for large model training. However, the current model training efficiency of GPU cluster networks for large model training is low. Summary of the invention

[0003] The purpose of this application is to provide a cluster network control method and device, including the following technical solutions:

[0004] A first aspect of the present application provides a cluster network control method, comprising:

[0005] In the process of each GPU server in the cluster network executing each round of large model training task, the interaction time consumed when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers is collected; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task;

[0006] Determine the target GPU cards that need to be optimized for interaction based on the collected interaction time between target GPU cards;

[0007] The transmission priority of the network packet of the target GPU card that needs to be interactively optimized is increased to reduce the interaction time of the target GPU card that needs to be interactively optimized.

[0008] In a possible implementation, each GPU server calls a collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in other GPU servers;

[0009] Collecting the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, including:

[0010] The execution time consumed by the collective communication library is collected as the interaction time consumed by the first target GPU card and the second target GPU card.

[0011] In a possible implementation, when each GPU server calls the collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in another GPU server, communication identification information uniquely representing the information interaction between the first target GPU card and the second target GPU card is added to a network packet transmitted between the first target GPU card and the second target GPU card;

[0012] Collecting the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, including:

[0013] collecting a duration of a target data flow in a network card to which the first target GPU card is connected as an interaction time consumption between the first target GPU card and the second target GPU card;

[0014] The network packets in the target data stream are network packets carrying the communication identification information.

[0015] In a possible implementation, collecting the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers includes:

[0016] When the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, the waiting time of the data receiving end waiting for the data of the data sending end is collected as the interaction time between the first target GPU card and the second target GPU card.

[0017] In a possible implementation, the method further includes:

[0018] Monitor whether each main switch in the cluster network is abnormal; wherein each main switch is connected to a GPU card in a different GPU server for executing the same subtask of the large model training task;

[0019] When an abnormal main switch is detected, the target GPU card that performs information interaction through the abnormal main switch is switched to perform information interaction through the standby interaction machine.

[0020] In a possible implementation, monitoring whether each main switch is abnormal includes:

[0021] Receive keep-alive information periodically sent by each main switch;

[0022] If no keep-alive information is received from any master switch within any period, a keep-alive information request is sent to any master switch;

[0023] If the response information sent by any of the main switches is not received, it is determined that any of the main switches is abnormal.

[0024] In a possible implementation, the method further includes:

[0025] Monitor the network traffic of each port of each main switch in the cluster network; wherein each main switch is connected to a GPU card in a different GPU server for executing the same subtask of the large model training task;

[0026] When the network traffic of any port is greater than the traffic threshold, part of the network traffic of the port is switched to be forwarded through the standby switch.

[0027] In a possible implementation, switching part of the network traffic of any port to be forwarded through a standby switch includes:

[0028] In order of the allocation ratio from low to high, the network traffic of any port is gradually reduced until the network of any port is less than or equal to the traffic threshold;

[0029] Wherein, each time the allocation ratio is determined, the network traffic corresponding to the allocation ratio in the network traffic of any port is allocated to the standby switch.

[0030] A second aspect of the present application provides a cluster network control device, including:

[0031] A collection module is used to collect the interaction time consumed when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers during the process of each round of large model training task executed by each GPU server in the cluster network; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task;

[0032] A determination module is used to determine the target GPU card that needs to be interactively optimized based on the collected interaction time between the target GPU cards;

[0033] The adjustment module is used to increase the transmission priority of the network packet of the target GPU card that needs to be interactively optimized, so as to reduce the interaction time of the target GPU card that needs to be interactively optimized.

[0034] A third aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the cluster network control method of the first aspect or any implementation of the first aspect.

[0035] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the cluster network control method of the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0037] Figure 1 An example diagram of the structure of a cluster network for large model training provided in this application;

[0038] Figure 2 A flowchart of an implementation of the cluster network control method provided in this application;

[0039] Figure 3 Another implementation flow chart of the cluster network control method provided in this application;

[0040] Figure 4 A structural example diagram of a cluster network with a backup link provided in this application;

[0041] Figure 5 Provided for this application Figure 4 An example diagram of the connection between a single GPU server and a backup switch in the cluster network shown;

[0042] Figure 6 A schematic diagram of the structure of the cluster network control device provided in this application. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0044] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0046] The cluster network for large model training is usually composed of multiple interconnected GPU servers, each of which is equipped with multiple GPU cards.

[0047] Among them, each of the multiple GPU servers is deployed with a large model to be trained, and different GPU cards in each GPU server deploy different sub-modules of the large model (the large model usually includes several layers, and different sub-modules of the large model can be different layers of the large model). In different GPU servers, the GPU cards deploy the large model in the same way. The large model deployed in the GPU server can be a large language model, a multimodal large model, or other large models, and this application does not make specific limitations on this.

[0048] In the process of training a large model using a training data set, each GPU server will perform the large model training task. Different GPU servers process different subsets of the training data set when performing the large model training task (each subset is recorded as a sub-training data set). Different GPU cards in the same GPU server are used to perform different subtasks of the large model training task.

[0049] The training of the large model is a multi-round iterative process. Each round of training includes: the large models in different GPU servers process different sub-training data sets of the training data set to obtain processing results; then the GPU servers will interact to update the parameters of the large model according to the processing results of the large model in each GPU server, and then perform the next round of training on the large model through the training data set.

[0050] During the training of a large model, the interaction between GPU servers includes but is not limited to: the GPU cards used to execute the same subtask in different GPU servers interact with each other. Based on this, in the cluster network, the GPU cards used to execute the same subtask in different GPU servers are connected to the same switch to be connected through the switch. In other words, the GPU servers are connected through switches, and each switch is connected to the GPU cards used to execute the same subtask in different GPU servers.

[0051] like Figure 1 As shown, a structural example diagram of a cluster network for large model training provided in an embodiment of the present application includes M GPU servers, each GPU server includes K GPU cards, the M GPU servers are connected through K switches, each switch is connected to M GPU cards, and the M GPU cards are GPU cards in different GPU servers for executing the same subtask. Figure 1 In the example shown, GPU cards with the same number in different GPU servers are used to execute the same subtask of the large model training task. For example, the i-th (i=1, 2, 3, ..., K) GPU card GPU-i in each GPU server is used to execute the same subtask of the large model training task, and the j-th (j=1, 2, 3, ..., K; i≠j) GPU card GPU-j in each GPU server is used to execute the same subtask of the large model training task. In the same server, the i-th GPU card GPU-i and the j-th GPU card GPU-j are used to execute different subtasks of the large model training task.

[0052] The research in this application found that during the training of large models, each round of iteration requires different GPU servers to synchronize information (for example, data and progress). The subtasks that are ahead in information synchronization need to wait for the subtasks that are behind in information synchronization. Only when all subtasks have completed information synchronization in each round of iterative training can the next round of iterative training begin. If the information synchronization of any subtask is not completed, the next round of iterative training cannot be started, resulting in low training efficiency of large models.

[0053] In order to improve the training efficiency of large models, this application scheme is proposed.

[0054] like Figure 2 As shown, a flowchart of an implementation of the cluster network control method provided in an embodiment of the present application may include:

[0055] Step S201: Collect the interaction time between target GPU cards used to execute the same subtask of the large model training task: During the process of each GPU server in the cluster network executing each round of large model training tasks, collect the interaction time when the target GPU card in each GPU server exchanges information with the target GPU cards in other GPU servers; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task.

[0056] During each round of training of the large model, each GPU server may collect the interaction time consumed when the i-th GPU card GPU-i in the GPU server interacts with the i-th GPU card GPU-i in other GPU servers.

[0057] In a cluster network for large model training, according to the topology algorithm used for communication between GPU cards for executing the same subtask in different GPU servers, the i-th GPU card GPU-i in each GPU server exchanges information with the i-th GPU card GPU-i in at least two other GPU servers.

[0058] As an example, GPU cards used to execute the same subtask in different GPU servers communicate with each other using a ring topology algorithm, that is, GPU cards used to execute the same subtask in different GPU servers are sequentially connected into a ring, so that each GPU card can only exchange information with the previous GPU card and the next GPU card in the ring structure. Based on this, each GPU server can collect the interaction time when the i-th GPU card in this GPU server interacts with the i-th GPU card in the next GPU server, and the interaction time when the i-th GPU card in this GPU server interacts with the i-th GPU card in the previous GPU server.

[0059] As an example, when GPU cards in different GPU servers for executing the same subtask communicate with each other using a non-ring topology algorithm, one implementation method is that any two GPU cards in different GPU servers for executing the same subtask can interact through a switch. Based on this, each GPU server can collect the interaction time when the i-th GPU card in the GPU server interacts with the i-th GPU card in any other GPU server. That is to say, assuming that there are M GPU servers in the cluster network, then corresponding to each GPU card, each GPU server can collect M-1 interaction times. Assuming that there are K GPU cards in each GPU server, the same GPU cards in different GPU servers perform the same subtask in the large model training task. Based on this, corresponding to the q-th (q=1, 2, 3,..., K) GPU card, the first GPU server can collect the interaction time when the q-th GPU card interacts with the q-th GPU card in the second GPU server, collect the interaction time when the q-th GPU card interacts with the q-th GPU card in the third GPU server, collect the interaction time when the q-th GPU card interacts with the q-th GPU card in the fourth GPU server, and so on, until the interaction time when the q-th GPU card interacts with the q-th GPU card in the M-th GPU server is collected.

[0060] Step S202: Determine the target GPU card that needs to be optimized for interaction according to the collected interaction time consumption between target GPU cards.

[0061] For any two target GPU cards, it can be determined whether the two target GPU cards are target GPU cards that need to be interactively optimized based on the interaction time consumed when the two target GPU cards perform information interaction.

[0062] The target GPU cards that need to be interactively optimized may include, but are not limited to: target GPU cards that need to improve transmission efficiency.

[0063] Optionally, the target GPU card that needs to be interactively optimized can be determined based on the length of the interaction time between the target GPU cards. As an example, if the interaction time between the first target GPU card in the first GPU server and the second target GPU card in the second GPU server is greater than the time threshold, the first target GPU card and the second target GPU card are determined to be target GPU cards that need to be interactively optimized. The first GPU server and the second GPU server are any two different GPU servers, and the first target GPU card and the second target GPU card are GPU cards in different GPU servers for executing the same subtask of the large model training task.

[0064] In the case where any two GPU cards used to execute the same subtask in different GPU servers can interact through the switch, if the interaction time between any target GPU card as a data sending end and each target GPU card as a data receiving end is greater than the duration threshold, it is determined that any target GPU card as a data sending end is abnormal; similarly, if the interaction time between any target GPU card as a data receiving end and each target GPU card as a data sending end is greater than the duration threshold, it is determined that any target GPU card as a data receiving end is abnormal. A prompt message can be output to indicate that there is a GPU card abnormality.

[0065] Step S203: increasing the transmission priority of the network packet of the target GPU card that needs to be interactively optimized, so as to reduce the interaction time of the target GPU card that needs to be interactively optimized.

[0066] Optionally, the transmission priority of the network packets of the target GPU card that needs interactive optimization can be adjusted to the highest, and the transmission priority of the network packets of the target GPU card that does not need interactive optimization can be lowered, so that the cluster network can give priority to transmitting the network packets of the target GPU card that needs interactive optimization, thereby improving the transmission efficiency of the network packets of the target GPU card that needs interactive optimization.

[0067] Optionally, each GPU server may send an optimization request to the controller, where the optimization request carries identification information of a target GPU card that needs to be interactively optimized. After receiving the optimization request sent by any GPU server, the controller updates the network flow table corresponding to any GPU server according to the identification information of the target GPU card that needs to be interactively optimized carried in the optimization request, so that the transmission priority of the network packets of the target GPU card that needs to be interactively optimized in the network flow table corresponding to any GPU server is the highest, and the transmission priority of the network packets of the target GPU card that does not need to be interactively optimized is lower than the transmission priority of the network packets of the target GPU card that needs to be interactively optimized.

[0068] The network flow table is a data structure or table used to record and manage network communication traffic, which records the network flow identifier and its corresponding forwarding port and transmission priority.

[0069] The controller may be one of the GPU servers among the multiple GPU servers, or may be a server independent of the multiple GPU servers and dedicated to control.

[0070] After any GPU server receives the updated network flow table, it will give priority to transmitting the network packets between the target GPU cards based on the transmission priority of the network packets between the target GPU cards in the updated network flow table, that is, the network packets with high transmission priority will be transmitted first, so as to give priority to ensuring the priority of the network packets transmitted by the target GPU cards that need to be interactively optimized.

[0071] For example, the network packet with the highest transmission priority is transmitted first. After the transmission of the network packet with the highest transmission priority is completed, the transmission of the network packet with the second highest transmission priority is started, and then the transmission of the network packet with the third highest transmission priority is started, and so on.

[0072] For another example, network packets with different transmission priorities have different scheduling frequencies, wherein a network packet with a higher transmission priority is scheduled more frequently, thereby increasing the transmission rate of the network packets of the target GPU card that needs to be interactively optimized.

[0073] The cluster network control method provided in the embodiment of the present application collects the interaction time when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers during the process of each round of large model training task being executed by each GPU server in the cluster network; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task; the target GPU card that needs to be interactively optimized is determined based on the collected interaction time between the target GPU cards; the transmission priority of the network packet of the target GPU card that needs to be interactively optimized is increased to reduce the interaction time of the target GPU card that needs to be interactively optimized, that is, the network transmission priority of the subtask data in the GPU card that needs to be interactively optimized is prioritized, so that the information interaction of the same subtask in different GPU servers can be synchronized in real time, thereby improving the efficiency of large model training.

[0074] In an optional embodiment, during the execution of each round of large model training tasks, in order to accelerate the communication between GPU cards, each GPU server can call the collective communication library to implement information interaction between the first target GPU card in the GPU server and the second target GPU card in other GPU servers.

[0075] The Collective Communication Library is designed for parallel and distributed computing environments to achieve efficient data exchange and synchronization operations. The Collective Communication Library is a software library that provides collective communication operations, which are usually used for data exchange between a group of processes or nodes. These libraries reduce latency, increase bandwidth, and support a variety of hardware connections and transmission protocols by optimizing communication algorithms and paths. The Collective Communication Library may include, but is not limited to, the following communication primitives: Broadcast, Reduce, Allreduce, Scatter, and Allscatter. Different communication primitives are used to implement different collective communication operations.

[0076] As an example, the collective communication library may be any of the following collective communication libraries: NCCL (NVIDIA Collective Communications Library), Open MPI (Open Message Passing Interface), Gloo, HCCL (Huawei Collective Communication Library), and the like.

[0077] Accordingly, the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers can be collected in the following manner:

[0078] The execution time of the collective communication library is collected as the interaction time between the first target GPU card and the second target GPU card.

[0079] When communication between GPU cards for executing the same subtask in different GPU servers is achieved by calling the collective communication library, the time spent on calling the collective communication library to complete the information interaction between the first target GPU card and the second target GPU card (i.e., the execution time of the collective communication library) can be used as the interaction time between the first target GPU card and the second target GPU card.

[0080] Each GPU server can record the start time of calling the collective communication library and the end time of the information interaction between the first target GPU card and the second target GPU card after calling the collective communication library, and use the duration from the start time to the end time as the time consumed for calling the collective communication library to complete the information interaction between the first target GPU card and the second target GPU card.

[0081] In an optional embodiment, when each GPU server calls the collective communication library to implement information interaction between the first target GPU card in the GPU server and the second target GPU card in other GPU servers, the communication identification information that uniquely represents the information interaction between the first target GPU card and the second target GPU card is added to the network packet transmitted between the first target GPU card and the second target GPU card.

[0082] Optionally, the communication identification information may include but is not limited to the following: communication ID (communicatorID), source process ID, target process ID, communication primitive, algorithm type and tensor type, etc.

[0083] The communication ID is determined by negotiation between the GPU server and the collective communication library, and is used to uniquely identify the information interaction between the two GPU cards. The source process ID refers to the ID of the process in the GPU card that is the information sender. The target process ID refers to the ID of the process in the GPU card that is the information receiver. The communication primitive refers to the communication primitive used when calling the collective communication library. The algorithm type and tensor type refer to the algorithm type and tensor type when executing the collective communication operation corresponding to the communication primitive.

[0084] Accordingly, the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers can be collected in the following manner:

[0085] The duration of the target data flow in the network card to which the first target GPU card is connected is collected as the interaction time consumption between the first target GPU card and the second target GPU card.

[0086] The network packets in the target data flow are network packets carrying the above communication identification information, that is, network packets carrying the same communication identification information constitute a data flow.

[0087] The GPU server includes the same number of network cards as the number of GPU cards, and each GPU card is connected to the switch through a network card. Therefore, the network packets sent by each GPU card will be sent to the switch through the connected network card, and the switch will forward them to the GPU cards in other GPU servers. The data carrying capacity of the network packet is certain, and the amount of data sent by the GPU card is usually relatively large. Therefore, the data to be sent by the GPU card will be split into multiple data blocks and sent out through multiple network packets. The multiple data packets constitute a data stream.

[0088] As mentioned above, each GPU card will exchange information with at least two GPU cards. Therefore, at least two data streams will be transmitted in the network card connected to each GPU card. The data packet of the present application carries communication identification information for uniquely identifying an information interaction between two target GPU cards. Therefore, the network card can identify different data streams based on the communication identification information.

[0089] The duration of a data stream can be determined by the sending timestamp of the first network packet of the data stream and the sending timestamp of the last network packet, that is, the duration between the sending timestamp of the last network packet in the data stream and the sending timestamp of the first network packet in the data stream is the duration of the data stream. Based on this, the network card can parse the network packet to obtain the connection information of the transport layer. The connection information of the transport layer may include but is not limited to: communication node identification (for example, RDMA IP) and queue pair (Queue Pair, QP) number, number of transmissions (sequence number of network packet), network packet size, sending timestamp of network packet, etc. After parsing the sending timestamp of the network packet carrying the above communication identification information, the duration of the data stream composed of the network packets carrying the above communication identification information can be calculated through the sending timestamp of the first network packet carrying the above communication identification information and the sending timestamp of the last network packet carrying the above communication identification information.

[0090] In an optional embodiment, when the target GPU cards communicate with each other using a ring topology algorithm, each GPU card only interacts with its direct neighbors (for ease of description, recorded as the previous GPU card and the next GPU card). Specifically, each GPU card can only receive information sent by its previous GPU card and can only send information to the next GPU card. For example, taking the implementation of the ring algorithm by calling the allreduce primitive in the collective communication library as an example, after each GPU card receives the data block sent by the previous GPU card, it performs a reduction operation on the local data block, and then passes the reduction result to the next GPU card, and so on. In actual applications, the receiver must first prepare the receiving buffer and notify the sender before data transmission. Therefore, there is an implicit receiver-driven scheduling logic. Based on this, the interaction time when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers can be collected in the following way:

[0091] The waiting time of the data receiving end waiting for data from the data sending end when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers is collected as the interaction time between the first target GPU card and the second target GPU card.

[0092] If the first target GPU card is the data sender and the second target GPU card is the data receiver, the second target GPU card can start timing after preparing the receiving buffer and notifying the first target GPU card to send data, and use the timing result after receiving the data sent by the first target GPU card as the waiting time for waiting for data from the data sender.

[0093] Similarly, if the second target GPU card is the data sender and the first target GPU card is the data receiver, the first target GPU card can start timing after preparing the receiving buffer and notifying the second target GPU card to send data, and use the timing result after receiving the data sent by the second target GPU card as the waiting time for waiting for data from the data sender.

[0094] Furthermore, if the waiting time of the data receiving end for the data from the data sending end is longer than the preset time, it is considered that the data sending end is internally slow (for example, the calculation is slow, or the data loading is slow, etc.) or the communication between the data receiving end and the data receiving end is slow, and it is determined that the data sending end and the data receiving end are target GPU cards that need to be interactively optimized.

[0095] Furthermore, if the data receiving end still has not received the data sent by the data sending end when the timing time reaches the preset time, it is also considered that the data sending end is slow internally (for example, calculation is slow, or data loading is slow, etc.) or the communication between the data receiving end and the data receiving end is slow, and it is determined that the data sending end and the data receiving end are target GPU cards that need to be interactively optimized.

[0096] In an optional embodiment, when the target GPU cards communicate with each other using a ring topology algorithm, two methods (referred to as a first collection method and a second collection method) may be used to collect the interaction time consumed when the target GPU cards perform information interaction.

[0097] Among them, the first collection method is: when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, the execution time of the collective communication library when calling the collective communication library to realize the information interaction between the first target GPU card and the second target GPU card is collected. The specific implementation method can refer to the aforementioned embodiment, which will not be repeated here; or, the first collection method is: when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, the duration of the target data flow composed of network packets carrying the same communication identification information in the network card connected to the first target GPU card is collected. The specific implementation method can refer to the aforementioned embodiment, which will not be repeated here.

[0098] The second collection method is: when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, the waiting time for the data receiving end to wait for the data from the data sending end is collected when the first target GPU card interacts with the second target GPU card. The specific implementation method can refer to the aforementioned embodiment and will not be repeated here.

[0099] When two methods are used to collect the interaction time consumption when information is interacted between target GPU cards, one implementation method for determining the target GPU card that needs to be interactively optimized may be:

[0100] First, the target GPU card that needs to be interactively optimized is determined according to the interaction time consumption between the target GPU cards collected by the first collection method.

[0101] As an example, if the execution time of the collective communication library when calling the collective communication library to implement information interaction between the first target GPU card and the second target GPU card is greater than the first duration threshold, it is considered that the communication between the first target GPU card and the second target GPU card is slow, and the first target GPU card and the second target GPU card are determined to be target GPU cards that need to be optimized for interaction.

[0102] As an example, if the duration of the target data flow composed of network packets carrying the same communication identification information in the network card connected to the first target GPU card is greater than the second duration threshold, it is considered that the communication between the first target GPU card and the second target GPU card is slow, and the first target GPU card and the second target GPU card are determined to be target GPU cards that need to be interactively optimized.

[0103] The target GPU card that needs to be optimized for interaction, determined based on the interaction time consumption collected by the first collection method, is a GPU card with slow communication.

[0104] For target GPU cards that do not require interaction optimization determined based on the interaction time between target GPU cards collected using the first collection method, target GPU cards that require interaction optimization can be determined based on the interaction time between target GPU cards collected using the second collection method.

[0105] As an example, if the waiting time for the target GPU card as the data receiver to wait for data from the target GPU card as the data sender is greater than the third time threshold, it is considered that the internal processing of the data receiver is slow, and the data receiver and the data sender are determined to be target GPU cards that need to be interactively optimized.

[0106] The target GPU card that needs to be interactively optimized, determined based on the interaction time collected through the second collection method, is a target GPU card that is not slow in communication but slow inside the data sending end.

[0107] In an optional embodiment, a backup link may also be set in the cluster network. For the sake of distinction, the aforementioned switches are referred to as master switches. The backup links include:

[0108] A controller connected to each main switch, a backup switch connected to the controller, and the backup switch is also connected to each GPU server.

[0109] The controller may be one of the GPU servers in the cluster network, or may be a server independent of the GPU servers and dedicated to control.

[0110] In order to prevent the GPU cards connected to the same switch from being unable to communicate when a switch fails, the cluster network control method provided by the present application may further include:

[0111] Monitor whether each main switch in the cluster network is abnormal; each main switch is connected to the GPU cards of the same subtask in different GPU servers for executing large model training tasks.

[0112] The controller can monitor whether each main switch in the cluster network is abnormal.

[0113] Optionally, in order to monitor each main switch, in this application, each main switch periodically sends keep-alive information to the controller. Specifically, each main switch can be monitored for abnormality in the following manner:

[0114] The controller receives keep-alive information periodically sent by each master switch.

[0115] If the keep-alive information sent by any master switch is not received within any period, a keep-alive information request is sent to the master switch.

[0116] The controller may periodically determine whether it has received the keep-alive information sent by each main switch. Assuming that the period for the main switch to send the keep-alive information is T, for each main switch, if after receiving the keep-alive information sent by the main switch, the controller does not receive the keep-alive information sent by the main switch within the time length T, it indicates that the main switch may be abnormal, and then the controller sends a keep-alive information request to the main switch to request the main switch to send the keep-alive information to the controller.

[0117] If the response information sent by any of the main switches is not received, it is determined that any of the main switches is abnormal.

[0118] After sending a keep-alive information request to the any one of the interchange machines, if the keep-alive information sent by the any one of the main interchange machines is still not received, it can be determined that the any one of the main interchange machines is abnormal.

[0119] When an abnormal main switch is detected, the target GPU card that performs information interaction through the abnormal main switch is switched to perform information interaction through the standby interaction machine.

[0120] Optionally, after detecting an abnormal main interaction machine, the controller can send control instructions to each GPU server to instruct each GPU server to connect the target GPU card connected under the abnormal main switch to the backup switch, and update the network flow table so that the GPU server can realize the interaction between the target GPU cards connected to the backup switch according to the updated network flow table.

[0121] By setting up a backup link and monitoring whether each main switch is abnormal, when a main switch is detected to be abnormal, the target GPU card that exchanges information through the abnormal main switch can be switched to exchange information through the backup interaction machine, so as to quickly restore the communication between the GPU cards connected to the abnormal main switch.

[0122] In the case of an abnormal main switch, the GPU cards connected to the abnormal main switch need to switch to the backup switch, which will cause the information interaction between the GPU cards connected to the abnormal main switch to lag behind the information interaction between the GPU cards connected to the normal main switch. Based on this, in an optional embodiment, when collecting the interaction time when the target GPU card in each GPU server interacts with the target GPU card in other GPU servers, only the interaction time when the target GPU card connected to the backup switch in each GPU server interacts with the target GPU card connected to the backup switch in other GPU servers can be collected. In other words, the interaction between the GPU cards under the backup switch is optimized first to ensure the network transmission priority and bandwidth of the subtasks with lagging information interaction.

[0123] When the GPU cards connected to the abnormal main switch are connected to the abnormal main switch, after completing one round of large model training tasks, when the GPU servers in the cluster network execute the next round of large model training tasks, the interaction time between each target GPU card connected to each main switch and the interaction time between each target GPU card connected to the backup switch can be collected.

[0124] In the above-mentioned embodiment, the backup link is used for fault recovery. In another embodiment, the backup link can be used as a supplementary link to share the traffic load of the main switch and maximize the overall cluster network bandwidth utilization. Based on this, in an optional embodiment, when a backup link is set in the cluster network, the cluster network control method provided in the embodiment of the present application may also include:

[0125] Monitor the network traffic of each port of each main switch in the cluster network; each main switch is connected to GPU cards in different GPU servers for executing the same subtask of the large model training task, and each port of the switch is connected to a GPU card.

[0126] Optionally, each main switch may count the network traffic of each port and actively report the network traffic to the controller, or the controller may actively request the switch to send the network traffic of each port.

[0127] When the network traffic of any port is greater than the traffic threshold, part of the network traffic of any port is switched to be forwarded through the standby switch to achieve load sharing.

[0128] If the network traffic of any port is greater than the traffic threshold, it means that the traffic load of any port is too large, and the backup switch can share part of the traffic of any port to reduce the traffic load of any port in the main switch.

[0129] When allocating network traffic to the standby switch, the network traffic of any port may be gradually reduced until the network flow of any port is less than or equal to the traffic threshold.

[0130] Specifically, the network traffic of any port may be gradually reduced in order of allocation ratio from low to high, until the network traffic of any port is less than or equal to the traffic threshold.

[0131] Wherein, each time the allocation ratio is determined, the network traffic corresponding to the allocation ratio in the network traffic of any port is allocated to the standby switch.

[0132] As an example, 10% of the network traffic of any port can be allocated to the backup switch first, and then the network traffic of any port can be continued to be counted. If the latest counted network traffic of any port is still greater than the traffic threshold, 10% of the latest network traffic of any port can be continued to be allocated to the backup switch, and then the network traffic of any port can be continued to be counted. If the latest counted network traffic of any port is less than or equal to the traffic threshold, it is determined that the traffic load sharing is completed and the load sharing process ends.

[0133] In an optional embodiment, the backup link can be used for fault recovery when a main switch is abnormal, and for load sharing when all main switches are normal. Based on this, another implementation flow chart of the cluster network control method provided in the embodiment of the present application is as follows: Figure 3 As shown, it may include:

[0134] Step S301: Monitor whether each main switch in the cluster network is abnormal; each main switch is connected to the GPU card of the same subtask in different GPU servers for executing the large model training task. If each main switch is normal, proceed to step S302; if an abnormal main switch is detected, proceed to step S303.

[0135] Step S302: Entering load sharing mode:

[0136] The network traffic of each port of each main switch in the cluster network is monitored, and when the network traffic of any port is greater than the traffic threshold, part of the network traffic of any port is switched to be forwarded through the standby switch.

[0137] Step S303: Entering the fault recovery mode:

[0138] The target GPU card that is performing information interaction through the abnormal main switch is switched to performing information interaction through the backup interaction machine.

[0139] The interaction time consumed when the target GPU card connected to the standby switch in each GPU server interacts with the target GPU cards connected to the standby switch in other GPU servers is collected.

[0140] According to the interaction time consumption between the target GPU cards connected to the standby switch, the target GPU cards connected to the standby switch that need to be interactively optimized are determined.

[0141] Improve the transmission priority of the network packets of the target GPU card that needs interactive optimization to reduce the interaction time of the target GPU card that needs interactive optimization.

[0142] Optionally, when an abnormal main switch is detected, if the standby switch carries the traffic of the normal main switch, you can first exit the load sharing mode, that is, return the network traffic shared by the standby switch to the normal main switch. After entering the fault recovery mode, the standby switch is only used for the interaction of the GPU card under the abnormal switch.

[0143] Furthermore, in the fault recovery mode, after one round of large model training tasks is completed and the next round of large model training tasks begins, in addition to collecting the interaction time of information interaction between target GPU cards connected to the backup switch, the interaction time of information interaction between target GPU cards connected to each main switch can also be collected to achieve synchronization of information interaction in the entire cluster network.

[0144] The solution of the present application is described below with reference to specific examples.

[0145] like Figure 4-5 As shown, Figure 4 A structural example diagram of a cluster network with a backup link provided in an embodiment of the present application, Figure 5 Provided for the embodiments of this application Figure 4 The following is an example diagram of how a single GPU server is connected to a backup switch in a cluster network.

[0146] When all main switches are normal, each GPU card in each GPU server is connected to the main switch through the switch chip and the main network card, for example:

[0147] GPU card No. 1 GPU-1 in GPU server-1 is connected to the main switch-1 through switch chip-1 and main network card-1;

[0148] GPU card No. 2 GPU-2 in GPU server-1 is connected to the main switch-2 through switch chip-1 and main network card-2;

[0149] GPU card No. 3 GPU-3 in GPU server-1 is connected to the main switch-3 through switch chip-1 and main network card-3;

[0150] GPU card No. 4 GPU-4 in GPU server-1 is connected to the main switch-4 through switch chip-1 and main network card-4;

[0151] GPU card No. 5 GPU-5 in GPU server-1 is connected to the main switch-5 through switch chip-2 and main network card-5;

[0152] GPU card No. 6 GPU-6 in GPU server-1 is connected to the main switch-6 through switch chip-2 and main network card-6;

[0153] GPU card No. 7 GPU-7 in GPU server-1 is connected to the main switch-7 through switch chip-2 and main network card-7;

[0154] GPU card No. 8 GPU-8 in GPU server-1 is connected to main switch-8 through switch chip-2 and main network card-8.

[0155] Each main switch may be a single switch or a group of switches, and the group of main switches forms a cascade structure.

[0156] The primary network card can be an Ethernet interface card (NIC) or other types of network cards.

[0157] In the above example, each GPU server is a GPU server including dual-channel CPU nodes (CPU cards). The CPU nodes are connected through a QPI bus or a UPI bus. Each CPU node is connected to a switching chip (PCIe Switche, also called a switching node) through an RC (Root Complex, i.e., the root node of the PCIe device tree). The interaction chip here can be a single interaction chip or a group of switching chips, and the group of switching chips forms a cascade structure.

[0158] The present application may also be used in a GPU server including a single-socket CPU node, or may be used in a GPU server including a quad-socket CPU node.

[0159] The backup network card can be a SmartNIC, which is also connected to the switch chip. The two backup network cards are connected through backup links (such as Figure 5 The backup links 1-1 and 1-2 in the

[0160] The controller may be a software defined network (SDN) controller, which controls the network through a control link (such as Figure 4 The dotted line in the figure is connected to the main switch and the backup switch to realize function recovery control when the main switch fails, and / or load sharing control when the main switches are normal. In addition, the backup switch can also forward the control traffic interacting between the controller and the GPU server through the backup link.

[0161] The backup switch can be a single switch supporting a small number of ports, or a cascaded set of multiple switches supporting a large number of ports.

[0162] Each GPU server (also called a worker node) is deployed with a collective communication library, and a training script (for example, a Python training script) and a control component (referred to as a Brain component) are deployed with a CPU node. The process of cluster network training a large model may include:

[0163] Each GPU server obtains training data. The training data obtained by each GPU server can be obtained by one of the GPU servers, which then groups the input training data and distributes it to each GPU server; or, the training data obtained by each GPU server can be obtained by a GPU server that is specially scheduled to receive the input training data, which then groups the input training data and distributes it to each GPU server used for large model training.

[0164] The training script in each GPU server schedules each GPU card in the GPU server to perform forward processing on the obtained training data, and then perform gradient back propagation. When performing gradient back propagation, the training script in each GPU server calls each GPU card to synchronize and reduce the gradient data between GPU cards that execute the same subtask, until the gradient data reduction of all GPU cards in each GPU server is completed; wherein, when synchronizing the gradient data between GPU cards that execute the same subtask, the training script in each GPU server calls the collective communication library (for example, the enhanced collective communication library Enhanced NCCL) to realize the synchronous interaction of the gradient data between GPU cards that execute the same subtask. The specific training process can be referred to the existing solution, which will not be described in detail here.

[0165] In order to improve the efficiency of large model training in cluster networks, this application proposes the following control strategy:

[0166] During each round of large model training, the training script in each GPU server reports the iteration round to the Brain component. The Brain component writes the received iteration round into the environment variable ITER_ROUND so that the GPU card that needs to be interactively optimized can be determined based on the data of the same iteration round.

[0167] When calling the collective communication library, the training script will record the start time of calling the collective communication library. When the collective communication library is executed, it indicates that the communication between the two GPU cards has ended. The time from the start of calling the collective communication library to the end of the collective communication library execution is the interaction time between the two GPU cards, which is recorded as the execution time of the collective communication library. The training script can also associate the communication operator with the iteration round. The training script reports the communication operator and the execution time when calling the collective communication library to the Brain component. The training script can also report the start time of calling the collective communication library to the Brain component. The Brain component can associate the communication operator with the iteration round, and can judge whether the two GPU cards associated with the communication operator are GPU cards that need to be optimized for interaction based on the execution time associated with the communication operator. The specific implementation method can refer to the aforementioned embodiment, which will not be repeated here.

[0168] Alternatively, when calling the collective communication library, the training script combines the six features tracked by the collective communication library into a communication operator as the communication identification information that uniquely identifies the interaction between the two GPU cards, and associates the communication operator with the iteration round, and adds the communication operator to the network packet carrying gradient information sent by the source process. The network card connected to the GPU card can parse the network packet to obtain the connection information of the transport layer, such as RDMA IP and QP number, number of transmissions (i.e., the serial number of the network packet), transmission packet size, sending timestamp and other information. The network card can also parse the communication operator in the network packet and associate it with the connection information of the transport layer, and send it to the Brain component. The Brain component can calculate the interaction time between the two GPU cards associated with the communication operator by associating the sending timestamp of the same communication operator, and judge whether the two GPU cards associated with the communication operator are GPU cards that need to be optimized for interaction based on the interaction time. The specific implementation method can refer to the aforementioned embodiment, which will not be repeated here.

[0169] When GPU cards executing the same subtask communicate with each other using a ring topology algorithm, the GPU card as the data receiving end can monitor the time it takes to wait for data to be sent by the data sending end. The GPU card determines the waiting time as the interaction time between the GPU card as the data receiving end and the GPU card as the data sending end and reports it to the Brain component.

[0170] The implementation method of the Brain component determining the target GPU card that needs to be interactively optimized based on the interaction time consumption can be found in the aforementioned embodiment and will not be repeated here.

[0171] After determining the target GPU card that needs to be interactively optimized, the Brain component sends an optimization request to the controller to request the controller to update the network flow table. The optimization request sent by the Brain component to the controller carries the identification information of the target GPU card that needs to be interactively optimized. The controller increases the transmission priority of the target GPU card that needs to be interactively optimized according to the identification information of the target GPU card that needs to be interactively optimized, updates the network flow table, and returns the updated network flow table to the Brain component. The Brain component configures the transmission priority of the network card connected to the target GPU card that needs to be interactively optimized according to the updated network flow table, so that the network card connected to the target GPU card that needs to be interactively optimized forwards network packets based on the transmission priority of each target GPU card in the updated network flow table.

[0172] In another implementation, each main switch periodically sends keep-alive information to the controller. If the controller does not receive keep-alive information sent by a main switch (for ease of description, recorded as the first main switch) within a certain period, the controller sends a request message to the first main switch. Afterwards, if the first main switch returns the keep-alive information to the controller, it is considered that the first main switch is normal. If the first main switch does not return the keep-alive information to the controller, it is considered that the first main switch is abnormal. At this time, the controller can send control instructions to each GPU server through the control link to instruct each GPU server to switch each target GPU card connected to the first main switch to the backup switch, collect the interaction time of each target GPU card connected to the backup switch when performing information interaction, and determine the target GPU card that needs to be optimized for interaction (see the previous section for the specific implementation method). The embodiment described above will not be repeated here), an optimization request is sent to the controller to request the controller to update the network flow table, the optimization request sent to the controller carries the identification information of the target GPU card that needs to be interactively optimized, the controller increases the transmission priority of the target GPU card that needs to be interactively optimized according to the identification information of the target GPU card that needs to be interactively optimized, and updates the network flow table, and returns the updated network flow table to the Brain component in the GPU server, and the Brain component configures the transmission priority of the standby network card according to the updated network flow table, so that the standby network card forwards the network packet based on the transmission priority of each target GPU card in the updated network flow table, so as to ensure that the standby network card preferentially transmits the network packet of the target GPU card that needs to be interactively optimized.

[0173] If the controller receives the keep-alive information sent by each main switch in each cycle, it is considered that each main switch is normal. At this time, a control instruction can be sent to each main switch through the control link to instruct each main switch to monitor the network traffic of each port. If the network traffic of any port is greater than the traffic threshold, a control instruction can be sent to the GPU server where the GPU card connected to any port is located through the control link to instruct the GPU server to switch part of the traffic of the GPU card connected to any port to the backup switch through the backup network card to achieve traffic sharing.

[0174] Corresponding to the method embodiment, the present application also provides a cluster network control device. A structural schematic diagram of the cluster network control device provided by the present application is as follows: Figure 6 As shown, it may include:

[0175] Collection module 601, determination module 602 and adjustment module 603;

[0176] The collection module 601 is used to collect the interaction time consumed when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers during the process of each round of large model training task executed by each GPU server in the cluster network; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task;

[0177] The determination module 602 is used to determine the target GPU card that needs to be interactively optimized according to the collected interaction time consumption between the target GPU cards;

[0178] The adjustment module 603 is used to increase the transmission priority of the network packet of the target GPU card that needs to be interactively optimized, so as to reduce the interaction time consumption of the target GPU card that needs to be interactively optimized.

[0179] The cluster network control device provided by the present application collects the interaction time when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers during the process of each round of large model training task being executed by each GPU server in the cluster network; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task; the target GPU cards that need to be interactively optimized are determined based on the collected interaction time between the target GPU cards; the transmission priority of the network packets of the target GPU cards that need to be interactively optimized is increased to reduce the interaction time of the target GPU cards that need to be interactively optimized, that is, the network transmission priority of the subtask data in the GPU cards that need to be interactively optimized is prioritized, so that the data of the same subtask in different GPU servers can be synchronized in real time, thereby improving the efficiency of large model training.

[0180] In an optional embodiment, each GPU server calls a collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in other GPU servers;

[0181] The collecting module 601 collects the time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, and is used to:

[0182] The execution time consumed by the collective communication library is collected as the interaction time consumed by the first target GPU card and the second target GPU card.

[0183] In an optional embodiment, when each GPU server calls the collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in another GPU server, communication identification information uniquely representing the information interaction between the first target GPU card and the second target GPU card is added to a network packet transmitted between the first target GPU card and the second target GPU card;

[0184] The collecting module 601 collects the time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, and is used to:

[0185] collecting a duration of a target data flow in a network card to which the first target GPU card is connected as an interaction time consumption between the first target GPU card and the second target GPU card;

[0186] The network packets in the target data stream are network packets carrying the communication identification information.

[0187] In an optional embodiment, the collecting module 601 collects the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, and is used to:

[0188] When the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, the waiting time of the data receiving end waiting for the data of the data sending end is collected as the interaction time between the first target GPU card and the second target GPU card.

[0189] In an optional embodiment, the cluster network control device further includes:

[0190] The first control module is used to monitor whether each main switch in the cluster network is abnormal; wherein each main switch is connected to GPU cards of the same subtask for executing the large model training task in different GPU servers; when an abnormal main switch is detected, the target GPU card for information interaction through the abnormal main switch is switched to information interaction through a backup interaction machine.

[0191] In an optional embodiment, when the first control module monitors whether each main switch is abnormal, it is used to:

[0192] Receive keep-alive information periodically sent by each main switch;

[0193] If no keep-alive information is received from any master switch within any period, a keep-alive information request is sent to any master switch;

[0194] If the response information sent by any of the main switches is not received, it is determined that any of the main switches is abnormal.

[0195] In an optional embodiment, the cluster network control device further includes:

[0196] The second control module is used to monitor the network traffic of each port of each main switch in the cluster network; wherein each main switch is connected to the GPU cards of different GPU servers for executing the same subtask of the large model training task; when the network traffic of any port is greater than the traffic threshold, part of the network traffic of any port is switched to be forwarded through the backup switch.

[0197] In an optional embodiment, when the second control module switches part of the network traffic of any port to be forwarded through the standby switch, it is used to:

[0198] In order of the allocation ratio from low to high, the network traffic of any port is gradually reduced until the network of any port is less than or equal to the traffic threshold;

[0199] Wherein, each time the allocation ratio is determined, the network traffic corresponding to the allocation ratio in the network traffic of any port is allocated to the standby switch.

[0200] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any human-computer interaction method provided in the embodiment of the present application.

[0201] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any human-computer interaction method provided in the embodiment of the present application.

[0202] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0203] It should be understood that in the embodiments of the present application, the various embodiments and features can be combined with each other to solve the aforementioned technical problems.

[0204] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the technology can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0205] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0206] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0207] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0208] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cluster network control method, comprising: In the process of each GPU server in the cluster network executing each round of large model training task, the interaction time consumed when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers is collected; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task; Determine the target GPU cards that need to be optimized for interaction based on the collected interaction time between target GPU cards; The transmission priority of the network packet of the target GPU card that needs to be interactively optimized is increased to reduce the interaction time of the target GPU card that needs to be interactively optimized.

2. According to the method of claim 1, each GPU server calls a collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in other GPU servers; Collect the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, including: The execution time consumed by the collective communication library is collected as the interaction time consumed by the first target GPU card and the second target GPU card.

3. According to the method of claim 1, when each GPU server calls the collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in other GPU servers, communication identification information uniquely representing the information interaction between the first target GPU card and the second target GPU card is added to the network packet transmitted between the first target GPU card and the second target GPU card; Collect the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, including: collecting a duration of a target data flow in a network card to which the first target GPU card is connected as an interaction time consumption between the first target GPU card and the second target GPU card; The network packets in the target data stream are network packets carrying the communication identification information.

4. The method according to claim 1, collecting the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, comprising: When the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, the waiting time of the data receiving end waiting for the data of the data sending end is collected as the interaction time between the first target GPU card and the second target GPU card.

5. The method according to claim 1, further comprising: Monitor whether each main switch in the cluster network is abnormal; wherein each main switch is connected to a GPU card in a different GPU server for executing the same subtask of the large model training task; When an abnormal main switch is detected, the target GPU card that performs information interaction through the abnormal main switch is switched to perform information interaction through the standby interaction machine.

6. The method according to claim 5, monitoring whether each main switch is abnormal, comprises: Receive keep-alive information periodically sent by each main switch; If no keep-alive information is received from any master switch within any period, a keep-alive information request is sent to any master switch; If the response information sent by any of the main switches is not received, it is determined that any of the main switches is abnormal.

7. The method according to claim 1, further comprising: Monitor the network traffic of each port of each main switch in the cluster network; wherein each main switch is connected to a GPU card in a different GPU server for executing the same subtask of the large model training task; When the network traffic of any port is greater than the traffic threshold, part of the network traffic of the port is switched to be forwarded through the standby switch.

8. The method according to claim 7, wherein switching part of the network traffic of any port to be forwarded through the backup switch comprises: In order of the allocation ratio from low to high, the network traffic of any port is gradually reduced until the network of any port is less than or equal to the traffic threshold; Wherein, each time the allocation ratio is determined, the network traffic corresponding to the allocation ratio in the network traffic of any port is allocated to the standby switch.

9. A cluster network control device, comprising: A collection module is used to collect the interaction time consumed when the target GPU card in each GPU server interacts with the target GPU cards in other GPU servers during the process of each round of large model training task executed by each GPU server in the cluster network; the target GPU cards in different GPU servers are used to execute the same subtask of the large model training task; A determination module is used to determine the target GPU card that needs to be interactively optimized based on the collected interaction time between the target GPU cards; The adjustment module is used to increase the transmission priority of the network packet of the target GPU card that needs to be interactively optimized, so as to reduce the interaction time of the target GPU card that needs to be interactively optimized.

10. The device according to claim 9, wherein each GPU server calls a collective communication library to implement information interaction between a first target GPU card in the GPU server and a second target GPU card in other GPU servers; The collecting module collects the interaction time consumed when the first target GPU card in each GPU server interacts with the second target GPU card in other GPU servers, and is used to: The execution time consumed by the collective communication library is collected as the interaction time consumed by the first target GPU card and the second target GPU card.

Images