Distributed training method, device, equipment and system of model and computer program product

Abstract

The invention provides a distributed training method, device, equipment and system for a model and a computer program product, and the method comprises the steps: reading a historical networking link establishment file of a training cluster under the condition that a training task of a target model is submitted again; wherein the historical networking link establishment file is a file which is generated by the training cluster based on the historically submitted training task and is used for recording the communication relationship between the cards; and in the process that the target model carries out distributed training based on the training task, controlling each card to carry out communication according to the historical networking link establishment file. In the distributed training process of the target model based on the training task, each card can be directly controlled to communicate according to the historical networking link establishment file, so that the model training is quickly carried out, and the time consumption of the distributed training is shortened.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a distributed training method, device, equipment, system and computer program product for a model. Background Art

[0002] With the rapid development of artificial intelligence technology, machine learning models have shown significant advantages in dealing with complex problems. Among them, large models have attracted widespread attention due to their powerful representation capabilities. However, the training of large models usually requires huge computing resources, so distributed training has become an effective solution. In distributed training, a large machine learning model can be divided into multiple small parts and trained on different computing nodes in the training cluster. This method can significantly improve training efficiency and shorten training time.

[0003] However, existing distributed training systems face a major problem: once a training task is interrupted, the communication between the computing nodes needs to be re-coordinated. This process is very time-consuming for the training cluster, resulting in the inability to quickly carry out training after resubmitting the training task. Summary of the invention

[0004] Based on the above technical status, the present application proposes a distributed training method, device, equipment, system and computer program product for a model.

[0005] According to a first aspect of an embodiment of the present application, a distributed training method for a model is provided, the method comprising:

[0006] In the case of resubmitting the training task of the target model, reading the historical networking and link building file of the training cluster; wherein the historical networking and link building file is a file generated by the training cluster based on the training task submitted historically and used to record the communication relationship between the cards;

[0007] During the distributed training of the target model based on the training task, each card is controlled to communicate according to the historical networking and link building file.

[0008] Optionally, the training cluster includes multiple computing nodes, each of the computing nodes includes multiple cards, and the reading of the historical networking and link building files of the training cluster includes:

[0009] For each of the computing nodes, a target card is selected to read the historical networking and link building file; wherein the target card is any card in the computing node;

[0010] The read historical networking and chain building files are transmitted to the remaining cards of the computing nodes to which they belong through each of the target cards.

[0011] Optionally, the training cluster includes multiple computing nodes, each of which includes multiple cards. Before reading the historical networking and link building file of the training cluster, the method further includes:

[0012] In the event of an unrecoverable failure of the target object, a standby computing node or standby card is selected to replace the target object, and the historical networking and link building file is updated based on the selected standby computing node or standby card;

[0013] The target object includes a computing node or a card.

[0014] Optionally, selecting a standby computing node or a standby card to replace the target object includes:

[0015] Get constraints;

[0016] Filter objects that meet the constraint conditions from a spare device library; wherein the spare device library includes a plurality of spare computing nodes or a plurality of spare cards that are communicatively connected to the training cluster;

[0017] The target object is replaced based on an object that meets the constraint condition.

[0018] Optionally, after obtaining the constraint conditions, the method further includes:

[0019] In the case that there is no object meeting the constraint condition in the standby device library, determining a set of objects to be replaced, wherein the set of objects to be replaced includes all computing nodes or all cards under a target hardware device, and the target hardware device is a hardware device at a higher level than the target object;

[0020] All computing nodes or all cards in the set of objects to be replaced are replaced.

[0021] Optionally, in the case of resubmitting the training task of the target model, before reading the historical networking and link building file of the training cluster, the method further includes:

[0022] In the event that a task interruption occurs during the distributed training of the target model based on the training task, the training task of the target model is resubmitted.

[0023] Optionally, the communication relationship between the cards includes: identity identification of each card, and unique identification of each card under different parallel schemes;

[0024] The parallel scheme includes at least one of data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism.

[0025] Optionally, before controlling each card to communicate according to the historical networking and link building file, the method further includes:

[0026] Performing a global communication capability test on all cards in the training cluster;

[0027] When all cards in the training cluster have the communication capability, each card is controlled to communicate according to the historical networking and link establishment file.

[0028] According to a second aspect of an embodiment of the present application, a distributed training device for a model is provided, the device comprising:

[0029] A reading module, used to read the historical networking and link building file of the training cluster when resubmitting the training task of the target model; wherein the historical networking and link building file is a file generated by the training cluster based on the training task submitted historically and used to record the communication relationship between the cards;

[0030] The training module is used to control each card to communicate according to the historical networking and chain building file during the distributed training of the target model based on the training task.

[0031] According to a third aspect of an embodiment of the present application, a distributed training system for a model is provided, the system comprising: a plurality of computing nodes, each of the computing nodes comprising a plurality of cards; a shared storage space, each of the cards being able to access the shared storage space;

[0032] Wherein, in the case of resubmitting the training task of the target model in the distributed training system, the card is used to read the historical networking and linking file of the training cluster; wherein the historical networking and linking file is a file generated by the training cluster based on the training task submitted historically and used to record the communication relationship between the cards; the historical networking and linking file is stored in the shared storage space;

[0033] During the distributed training of the target model based on the training task, the card is used to communicate according to the historical networking and link building file.

[0034] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the distributed training method of the model as described in the first aspect by running the program in the memory.

[0035] According to a fifth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the distributed training method of the model described in the first aspect is implemented.

[0036] According to a sixth aspect of an embodiment of the present application, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the distributed training method of the model as described in the first aspect.

[0037] In an embodiment of the present application, when resubmitting the training task of the target model, there is no need to re-coordinate the communication between the computing nodes. Instead, the historical networking and chain building file of the training cluster is directly read. Since the historical networking and chain building file is a file that records the communication relationship between the cards generated by the training cluster based on the historically submitted training tasks. Therefore, the historical networking and chain building file can be regarded as the result of coordinating the communication between the computing nodes or cards in the training cluster. In the process of distributed training of the target model based on the training task, the communication between the cards can be directly controlled according to the historical networking and chain building file, thereby quickly carrying out model training and shortening the time of distributed training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of an application environment for a distributed training method of a model provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a flow chart of a distributed training method for a model provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a practical application of a distributed training method for a model provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of a distributed training device for a model provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] Overview

[0046] As described in the background technology, in the distributed training scenario of the model, especially the large model or the super large model, it is usually necessary to use a training cluster composed of multiple cards. Through reasonable division of labor and cooperation between multiple cards, the distributed training of the model can be completed. For example, in the distributed training process, the cards in the training cluster are usually divided into multiple communication groups based on parallel technology (such as data parallelism, pipeline parallelism, tensor parallelism, and sequence parallelism). The cards in the same communication group will communicate with each other to complete the distributed training of the model.

[0047] In order to ensure that the cards in the same communication group can communicate with each other, it is necessary to establish a communication link for the cards in the training cluster at the beginning of training, so that each card can know which cards belong to the same communication group and can communicate with them accurately. The process of establishing a communication link is generally: select a card in the communication group as the central node, which obtains the status information of all the remaining cards in the communication group, and then broadcasts the status information to all the remaining cards in the communication group.

[0048] However, during the above model training process, the task will always be interrupted due to various reasons. This requires resubmitting the training task and executing the above communication link establishment process again. Moreover, with the frequent occurrence of task interruptions, the entire training process will be time-consuming.

[0049] In view of the above-mentioned technical status, an embodiment of the present application proposes a distributed training method for a model. After the training task is interrupted, when the communication link is re-established, it is no longer necessary to coordinate the communication between the cards in the communication group. Instead, the historical networking and link establishment file of the training cluster is directly read. Since the historical networking and link establishment file is a file generated by the training cluster based on the historically submitted training tasks, it is used to record the communication relationship between the cards. Therefore, the historical networking and link establishment file can be regarded as the result of coordinating the communication between the computing nodes or cards in the training cluster. In the process of distributed training of the target model based on the training task, each card can be directly controlled to communicate according to the historical networking and link establishment file, thereby quickly carrying out model training and shortening the time consumption of distributed training.

[0050] Exemplary Implementation Environment

[0051] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the implementation environment involved in this application, but the implementation environment of the method provided by this application is not limited to Figure 1 The implementation environment shown. Figure 1The implementation environment includes: a parameter server 001, a computing node 002 communicating with the parameter server 001, wherein the computing node 002 includes a plurality of cards 003; wherein the number of the parameter server 001, the computing node 002 and the card 003 is not limited in this embodiment, Figure 1 Just an example.

[0052] Card 003 can be an independent hardware device on computing node 002, which can usually be a GPU, TPU or NPU. Card 003 has highly parallel computing capabilities and can accelerate the training process of the model. In distributed training, card 003 on each computing node 002 is responsible for performing a part of the training task, such as calculating gradients or updating model parameters. Computing node 002 is the basic unit in the implementation environment and is responsible for performing the training task of the model. Each computing node 002 can be a physical computer or a virtual machine with independent processor, memory and storage resources. During the training process, computing node 002 obtains model parameters from parameter server 001, performs training tasks, and sends the updated parameters back to parameter server 001. Parameter server 001 is a core component in the implementation environment and is responsible for storing and managing model parameters. Parameter server 001 is usually composed of a group of high-performance computers, which are connected through a network. During the training process, parameter server 001 is responsible for summarizing the updated data of each computing node 002 and updating the global model parameters. Figure 1 The platform built by the implementation environment shown can be regarded as a training cluster required for model distributed training. When the distributed training of the model is performed in the training cluster, the training can be performed based on the method provided in this application to shorten the duration of model training.

[0053] Exemplary Methods

[0054] See also Figure 2 In an exemplary embodiment, a distributed training method for a model is provided, which can be applied to Figure 1 In the implementation environment shown, but not limited thereto. The method provided in this application can be applied to all distributed training architectures that complete distributed training of models through training clusters. The method may include:

[0055] S201: When resubmitting the training task of the target model, read the historical networking and link establishment file of the training cluster.

[0056] In this step, the target model is the object of model training, which can be any machine learning model. For example, the target model can be a large model or a super large model, but is not limited thereto. Similarly, the training task can be a training task set based on any business needs, and this application does not limit its specific content.

[0057] Resubmitting the training task of the target model indicates that this training for the training task is not the first training. The task is interrupted due to some reasons, so it is necessary to resubmit the training task. In some embodiments, when resubmitting the training task of the target model, before reading the historical networking and link building files of the training cluster, the method also includes: when a task interruption occurs during the distributed training of the target model based on the training task, resubmitting the training task of the target model. Among them, the reasons for the task interruption may be abnormal node crash, abnormal network communication of the node link, abnormal device storage, etc., which are not limited in this application.

[0058] The historical networking and linking file is a file generated by the training cluster based on the training tasks submitted historically, and is used to record the communication relationship between each card. Since the historical networking and linking file records the communication relationship between each card. Therefore, it is possible to determine which cards in the training cluster need to communicate or which cards belong to the same communication group through the historical networking and linking file. The historical networking and linking file can be regarded as the result of coordinating the communication between each computing node or each card in the training cluster. In some embodiments, the historical networking and linking file can be generated during the communication linking process when the training task is submitted to the target model for the first time. Among them, the communication linking performed when the training task is submitted for the first time can adopt the existing communication linking scheme. That is, for each communication group, a card in the communication group is selected as the central node, which obtains the status information of all the remaining cards in the communication group, and then broadcasts the status information to all the remaining cards in the communication group.

[0059] In some embodiments, the historical networking link building file is stored in the shared storage space of the training cluster, so that each card can access / read the content of the historical networking link building file. The historical networking link building file can also be used as a node parallel description file, and the communication library required for the training process supports the reading and parsing of the node parallel description file.

[0060] S202: During the distributed training of the target model based on the training task, the cards are controlled to communicate according to the historical networking and chain building files.

[0061] In this step, since the historical networking and chain building file records the communication relationship between each card, during the distributed training process, it is only necessary to implement the communication between each card according to this file. It is worth noting that during the distributed training of the target model based on the training task, controlling the communication between each card according to the historical networking and chain building file can implement a parallel strategy for distributed training and improve training efficiency.

[0062] In an embodiment of the present application, when resubmitting the training task of the target model, there is no need to re-coordinate the communication between the computing nodes. Instead, the historical networking and chain building file of the training cluster is directly read. Since the historical networking and chain building file is a file that records the communication relationship between the cards generated by the training cluster based on the historically submitted training tasks. Therefore, the historical networking and chain building file can be regarded as the result of coordinating the communication between the computing nodes or cards in the training cluster. In the process of distributed training of the target model based on the training task, the communication between the cards can be directly controlled according to the historical networking and chain building file, thereby quickly carrying out model training and shortening the time of distributed training.

[0063] In some embodiments of the present application, the training cluster includes multiple computing nodes, each computing node includes multiple cards, and reading the historical networking and link building files of the training cluster includes:

[0064] For each computing node, select a target card to read the historical networking and chain building file; wherein the target card is any card in the computing node;

[0065] The read historical networking and chain building files are transmitted to the remaining cards of the computing nodes to which they belong through each target card.

[0066] It should be noted that when all cards in the training cluster read the historical networking and chain building files at the same time, a certain reading pressure will be generated, and the reading pressure increases with the increase of the scale of the training cluster. In order to avoid excessive reading pressure, other methods can be used to inform all cards in the training cluster of the historical networking and chain building files. For example, a small number of cards are selected to read the historical networking and chain building files at the same time, and then the small number of cards that read the historical networking and chain building files transfer the read files to the remaining cards. In some embodiments, a card can be selected in each computing node to read the historical networking and chain building files, and then the card that reads the historical networking and chain building files broadcasts the read data to the remaining cards under the same computing node. Assume that the training cluster has 1000 computing nodes, and each computing node contains 8 cards. In this embodiment, only 1000 cards are required to read the historical networking and chain building files at the same time, which undoubtedly reduces the reading pressure compared to 8000 cards reading the historical networking and chain building files at the same time.

[0067] In the embodiment of the present application, compared with all cards in the training cluster reading the historical networking and link building files at the same time, only a small number of cards need to read the historical networking and link building files at the same time, thereby reducing the reading pressure.

[0068] In order to improve the stability and robustness of the training cluster and avoid errors in the training process caused by partial hardware failure, in some embodiments of the present application, the training cluster includes multiple computing nodes, each computing node includes multiple cards, and before reading the historical networking and link building file of the training cluster, the method also includes:

[0069] In the event of an unrecoverable failure of a target object, a backup computing node or a backup card is selected to replace the target object, and the historical networking and chaining files are updated based on the selected backup computing node or the backup card; wherein the target object includes a computing node or a card.

[0070] It should be noted that an unrecoverable fault may refer to a fault that cannot be repaired or cannot be automatically repaired by the training cluster. In this regard, it is necessary to replace the hardware or target object that has an unrecoverable fault by replacement. Since there are computing nodes and cards in the training cluster, both of which may fail, the target object may be a computing node or a card. It is understandable that when selecting a spare computing node or a spare card to replace the target object, the selected object is different based on the target object. Specifically, in the case of an unrecoverable fault in the computing node (the target object is a computing node), a spare computing node is selected to replace the computing node; in the case of an unrecoverable fault in the card (the target object is a card), a spare card is selected to replace the card. Since the process of replacing a computing node is similar to the process of replacing a card, for the sake of ease of description, the replacement process is described here only by replacing a computing node as an example.

[0071] When replacing computing nodes, there are computing nodes that do not participate in model training in the training cluster or the training cluster is connected to computing nodes that do not participate in model training. Therefore, these computing nodes that do not participate in model training can be regarded as spare computing nodes, and one or more of them can be selected to replace the faulty computing nodes. For example, if a computing node has an unrecoverable fault, a spare computing node can be selected to replace it. Similarly, if two computing nodes have unrecoverable faults, two spare computing nodes can be selected to replace them.

[0072] It is understandable that due to changes in the computing nodes or cards in the training cluster, the data in the historical networking and chain building files will no longer be accurate, so the historical networking and chain building files need to be changed accordingly. For example, if computing node A is used to replace computing node B, then the information of computing node B in the historical networking and chain building files needs to be updated to the information of computing node A.

[0073] Of course, for recoverable faults, there is no need to replace the target object. You can directly read the historical networking and link building files and perform related operations according to the historical networking and link building files.

[0074] In some embodiments, the fault type of the target object can be determined and corresponding operations can be performed when the parallel strategy and hardware structure have not changed (such as the number of cards has not changed). Among them, the fault type includes: recoverable fault and unrecoverable fault. Parallel strategies include: data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, etc.

[0075] In an embodiment of the present application, when an unrecoverable failure occurs in a computing node or card in a training cluster, the faulty object can be replaced, thereby improving the stability and robustness of the training cluster and avoiding errors in the training process caused by partial hardware failure.

[0076] In some embodiments of the present application, selecting a standby computing node or a standby card to replace a target object includes:

[0077] Get constraints;

[0078] Filter objects that meet the constraint conditions from a spare device library; wherein the spare device library includes a plurality of spare computing nodes or a plurality of spare cards that are communicatively connected to the training cluster;

[0079] Replace the target object based on the objects that meet the constraints.

[0080] It should be noted that when selecting a spare computing node or spare card, there may be multiple spare computing nodes or spare cards to choose from. Among them, the multiple spare computing nodes or spare cards can be regarded as objects in the spare device library. In some embodiments, one or more of them can be selected at will to replace the target object. However, the communication distance between the selected spare computing node or spare card and the target object may be very far. In this way, although the training cluster can be guaranteed to work normally after the replacement, the communication efficiency will be very low. For this reason, some constraints can be formulated in advance, and when selecting spare computing nodes or spare cards, objects that meet the constraints are selected. By formulating reasonable constraints, the aforementioned low communication efficiency can be avoided. For example, the constraint can be a computing node under the same layer 2 switch. In this way, a computing node located under the same layer 2 switch as the target object can be selected to replace the target object, which greatly shortens the communication distance and improves the communication efficiency. Of course, the constraints are not limited to this, and the constraints can be set based on the requirements of the basic communication facilities and the communication library.

[0081] In the embodiment of the present application, some constraints can be set in advance, and when selecting a spare computing node or spare card to replace the target object, an object that meets the constraints is selected. By setting reasonable constraints, the spare computing node or spare card can be flexibly selected, and the problem of low communication efficiency after the target object is replaced can be avoided.

[0082] In order to avoid the situation where the target object cannot be replaced due to failure to select a suitable backup computing node or backup card, in some embodiments of the present application, after obtaining the constraint condition, the method further includes:

[0083] In the case that there is no object meeting the constraint condition in the spare device library, determining a set of objects to be replaced, wherein the set of objects to be replaced includes all computing nodes or all cards under the target hardware device, and the target hardware device is the upper-level hardware device of the target object;

[0084] Replace all computing nodes or all cards in the set of objects to be replaced.

[0085] It should be noted that since there is no object that meets the constraint conditions in the spare device library, the target object that has an unrecoverable fault cannot be replaced at this time, which will cause a series of problems in the model training process. In this regard, you can arbitrarily select a spare computing node or spare card to replace the target object, first of all, to ensure that the training cluster can work normally. Of course, since the selected spare computing node or spare card does not meet the constraint conditions, there will be low communication efficiency or other problems. Therefore, in this embodiment, the hardware of the same level as the target object can be replaced together with the target object.

[0086] For example, the hardware level can be divided into the first level of switch level, the second level of computing node level and the third level of card level. Then, if the target object is a card, all cards under the computing node to which the card belongs can be replaced. The computing node to which the card belongs is the upper-level hardware device of the card. If the target object is a computing node, all computing nodes under the switch to which the computing node belongs can be replaced. The switch to which the computing node belongs is the upper-level hardware device of the computing node.

[0087] In the embodiment of the present application, it is possible to avoid the situation where the target object cannot be replaced due to failure to select a suitable backup computing node or backup card.

[0088] In some embodiments of the present application, the communication relationship between the cards includes: the identity of each card, the unique identification of each card under different parallel schemes;

[0089] The parallel scheme includes at least one of data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and mixture-of-experts parallelism.

[0090] It should be noted that the card identity is the unique identifier of the card in the training cluster, for example, the identity can be RankID. The parallel scheme is equivalent to the parallel strategy in the above embodiment. Among them, data parallelism can be understood as: segmenting by data, copying the same model to multiple devices, and executing different data slices in parallel. Tensor parallelism can be understood as segmenting the parameters in the calculation layer to different devices, that is, intra-layer parallelism or intra-operator parallelism (Intra-operatorParallelism). Pipeline parallelism can be understood as segmenting the layers of the model and distributing each segment on different computing devices, so that the previous and next stages can work in a pipeline and batch manner. The specific contents of sequence parallelism and expert hybrid parallelism will not be described in detail here.

[0091] It can be understood that the number of communication groups in which each card participates is related to the number of parallel schemes. For example, the two can be equal. In this way, all cards in the training cluster will be divided into a communication group under each parallel scheme. For ease of understanding, a specific example is given below. Assume that the training cluster includes 2 nodes (Node0, Node1), each node includes 8 cards, and a total of 16 cards participate in the training. Among them, the cards in the training cluster are numbered 0, 1, 2, ..., 15, cards 0-7 are in Node0, and cards 8-15 are in Node1. The parallel schemes for training are pipeline parallelism with a parallelism of 2; tensor parallelism with a parallelism of 2; data parallelism with a parallelism of 4. Then, one of the possible communication relationships between the cards can be shown in Table 1 below:

[0092]

[0093] Table 1

[0094] In Table 1, RankID represents the card's identity, TP group number represents the card's unique identification under the tensor parallel scheme, PP group number represents the card's unique identification under the pipeline parallel scheme, and DP group number represents the card's unique identification under the data parallel scheme. It can be seen that for a card with an identity of 0, it participates in three communication groups. For the convenience of description in the following description, the card with an identity of N is described as card No. N. Then the three communication groups that card No. 0 participates in are: the communication group composed of card No. 0 and card No. 1 under the tensor parallel scheme; the communication group composed of card No. 0, card No. 2, card No. 4 and card No. 6 under the data parallel scheme; the communication group composed of card No. 0 and card No. 8 under the pipeline parallel scheme.

[0095] Therefore, the communication relationship between the cards in the training cluster can be determined by Table 1. In some embodiments, the historical networking and link establishment file records the communication relationship between the cards in the form of Table 1 above.

[0096] In the embodiment of the present application, the communication relationship between the cards includes: the identity identification of each card, and the unique identification of each card under different parallel schemes. Through the corresponding relationship between the identifications of each card, the communication relationship can be simply and clearly described, avoiding the historical networking and chain establishment files from being too complicated.

[0097] In order to avoid the situation that the training is affected by the failure of some cards before the training, based on the above embodiment, in the embodiment of the present application, before controlling each card to communicate according to the historical networking link establishment file, the method also includes:

[0098] Perform a global communication capability test on all cards in the training cluster;

[0099] When all cards in the training cluster have the ability to communicate, control the communication between the cards according to the historical networking and chain building files.

[0100] It should be noted that during the distributed training of the training cluster, each card will participate and communicate with other cards. If it is found during the training process that the card that originally had the communication capability in the historical networking and link building file cannot communicate, it will inevitably affect the progress of the training task. For this reason, before controlling each card to communicate according to the historical networking and link building file, the communication capability of each card can be verified in advance to confirm whether each card is reachable. In this embodiment, the global communication capability test includes: testing the communication capability of each card in the training cluster to determine whether each card has the communication capability or is reachable. For example, each card can broadcast messages to all cards in the training cluster and receive messages broadcast by all cards in the training cluster. The communication capability of itself is judged based on the messages broadcasted by itself and the messages received, but it is not limited to this.

[0101] If some cards in the training cluster do not have the communication capability or are unavailable, it is considered that these cards or the computing nodes to which the cards belong have failed. Then the fault type is determined, where the fault types include recoverable faults and unrecoverable faults. For unrecoverable faults, the method of replacing the faulty object in the aforementioned embodiment can be used to handle it. That is, in the event of an unrecoverable fault in the target object, a backup computing node or a backup card is selected to replace the target object, and the historical networking and chaining files are updated based on the selected backup computing node or backup card; where the target object includes a computing node or a card. In this regard, the embodiments of the present application will not be repeated.

[0102] In the embodiment of the present application, before controlling each card to communicate according to the historical networking and chain building file, the communication capabilities of all cards in the training cluster are judged, so as to avoid the situation where the training is affected by the failure of some cards before training.

[0103] Usually, a very large training cluster is required in the training process of a large model. Due to the large number of computing nodes and cards in the very large training cluster, the entire link building process is very time-consuming when using traditional networking and link building solutions. Regardless of the status of the task, as long as the training is restarted, the communication link building process is exactly the same. Therefore, once the task is frequently interrupted, the time spent on communication link building is unacceptable. The following uses this scenario as an example to illustrate the application process of the distributed training method of the model provided in this application. Figure 3 As shown, including:

[0104] S301: Training initialization, preparing for model training before the training cluster starts distributed training. For example, it may include environment construction (determining the communication protocol, total number of processes, etc.), model state synchronization (synchronizing the model state in the main card to the remaining cards), model parameter initialization, etc.

[0105] S302: Determine whether this training is the first training. In some embodiments, each training in the distributed training process requires submission of a training task, and the submission information of the training task can be recorded, so as to determine whether it is the first training, that is, whether it is the first submission of the training task, based on the recorded information. If so, execute S303, otherwise execute S306.

[0106] S303: Communication link establishment. For example, the cards in the training cluster are divided into different communication groups according to the parallel scheme. For each communication group, card No. 0 collects the status information of the remaining cards in the group, and then broadcasts the collected status information to the remaining cards, so that each card knows which cards are in the same communication group as it. In addition, RankTable information is generated based on the communication relationship between the cards in each communication group, and saved in the FileA file. Among them, the RankTable information is equivalent to the aforementioned Table 1 storing the communication relationship of each card, and the FileA file is equivalent to the aforementioned historical networking link establishment file, which will not be repeated here.

[0107] S304: Save the FileA file to the shared storage so that each card in the training cluster can access the FileA file.

[0108] S305: Start training, and during the distributed training process, control the cards to communicate with each other according to the communication relationship between the cards in FileA.

[0109] S306: Determine whether the node is updated. The node here can be a computing node or card in the training cluster. If so, execute S308, if not, execute S307. In some embodiments, when a training task occurs and it is detected that the parallel scheme and the number of nodes of the training have not changed after the task is resubmitted, it can be determined whether the node has failed and the type of failure. Among them, the failure types include recoverable failures and unrecoverable failures. In the event of an unrecoverable failure, the failed node needs to be updated. This process is similar to the case where an unrecoverable failure occurs in the target object in the above embodiment, and will not be repeated here.

[0110] S307: Each card in the training cluster reads FileA and then executes S305.

[0111] S308: Update the node information corresponding to the file FileA. This process is the same as the case where the target object is replaced after an unrecoverable failure occurs in the target object in the above embodiment, and will not be described in detail here.

[0112] In the above application scenarios, the method provided by this application can be used for processing, which will greatly shorten the time consumption. When a task is interrupted, the time consumption of the communication link establishment of the entire training cluster will be greatly reduced. Moreover, in the recovery phase, the time cost of link establishment is independent of the number of cards, and the original minute-level link establishment process can be reduced to seconds, which is of great significance to the acceleration of training tasks and the effective utilization of training clusters.

[0113] Exemplary Devices

[0114] Accordingly, the present application also provides a distributed training device for a model, see Figure 4 As shown, the device comprises:

[0115] The reading module 401 is used to read the historical networking and link building file of the training cluster when the training task of the target model is resubmitted; wherein the historical networking and link building file is a file generated by the training cluster based on the training tasks submitted historically and used to record the communication relationship between the cards;

[0116] The training module 402 is used to control the communication of each card according to the historical networking and link building file during the distributed training of the target model based on the training task.

[0117] In some embodiments, the training cluster includes multiple computing nodes, each computing node includes multiple cards, and the reading module 401 is specifically used to select a target card for each computing node to read the historical networking and chain building file; wherein the target card is any card in the computing node; and the read historical networking and chain building files are transmitted to the remaining cards of the computing nodes to which they belong through each target card.

[0118] In some embodiments, the training cluster includes a plurality of computing nodes, each computing node includes a plurality of cards, and the apparatus further includes:

[0119] The fault module is used to select a spare computing node or spare card to replace the target object when an unrecoverable fault occurs to the target object, and update the historical networking and chain building files based on the selected spare computing node or spare card; wherein the target object includes a computing node or a card.

[0120] In some embodiments, the fault module is specifically configured to:

[0121] Get constraints;

[0122] Filter objects that meet the constraint conditions from a spare device library; wherein the spare device library includes a plurality of spare computing nodes or a plurality of spare cards that are communicatively connected to the training cluster;

[0123] Replace the target object based on the objects that meet the constraints.

[0124] In some embodiments, the device further comprises:

[0125] The replacement module is used to determine a set of objects to be replaced when there is no object that meets the constraint conditions in the spare device library, wherein the set of objects to be replaced includes all computing nodes or all cards under the target hardware device, and the target hardware device is the upper-level hardware device of the target object; and replace all computing nodes or all cards in the set of objects to be replaced.

[0126] In some embodiments, the device further comprises:

[0127] The task interruption module is used to resubmit the training task of the target model when a task interruption occurs during the distributed training of the target model based on the training task.

[0128] In some embodiments, the communication relationship between the cards includes: the identity of each card, the unique identification of each card under different parallel schemes;

[0129] The parallel scheme includes at least one of data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism.

[0130] In some embodiments, the device further comprises:

[0131] The global communication module is used to test the global communication capability of all cards in the training cluster. When all cards in the training cluster have communication capability, the communication between the cards is controlled according to the historical networking and chain establishment files.

[0132] The distributed training device of the model provided in this embodiment belongs to the same application concept as the distributed training method of the model provided in the above embodiments of this application, and can execute the distributed training method of the model provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not described in detail in this embodiment, please refer to the specific processing content of the distributed training method of the model provided in the above embodiments of this application, which will not be repeated here.

[0133] It should be understood that the modules in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the remaining part is implemented in the form of a hardware circuit.

[0134] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0135] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0136] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0137] Exemplary Systems

[0138] Accordingly, the embodiment of the present application also provides a distributed training system for a model, and the distributed training system for the model includes:

[0139] A plurality of computing nodes, each of the computing nodes comprising a plurality of cards;

[0140] A shared storage space, to which each of the cards can access;

[0141] Wherein, when the training task of the target model is resubmitted in the distributed training system, the card is used to read the historical networking and chain building file of the training cluster; wherein, the historical networking and chain building file is a file generated by the training cluster based on the training task submitted historically, and is used to record the communication relationship between each card; the historical networking and chain building file is stored in the shared storage space; during the distributed training of the target model based on the training task, the card is used to communicate according to the historical networking and chain building file.

[0142] The distributed training system of the model provided in this embodiment belongs to the same application concept as the distributed training method of the model provided in the above embodiments of this application, and can execute the distributed training method of the model provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not described in detail in this embodiment, please refer to the specific processing content of the distributed training method of the model provided in the above embodiments of this application, which will not be repeated here.

[0143] Exemplary Electronic Devices

[0144] The present application embodiment provides an electronic device, see Figure 5 As shown, the device includes:

[0145] Memory 500 and processor 510;

[0146] The memory 500 is connected to the processor 510 and is used to store programs;

[0147] The processor 510 is used to implement the distributed training method of the model disclosed in any of the above embodiments by running the program stored in the memory 500.

[0148] Specifically, the electronic device may further include: a bus, a communication interface 520 , an input device 530 and an output device 540 .

[0149] The processor 510, the memory 500, the communication interface 520, the input device 530 and the output device 540 are connected to each other via a bus.

[0150] A bus may include a pathway that transfers information between components of a computer system.

[0151] The processor 510 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0152] The processor 510 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0153] The memory 500 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 500 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0154] The input device 530 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0155] Output device 540 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0156] The communication interface 520 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0157] The processor 510 executes the program stored in the memory 500 and calls other devices, which can be used to implement the various steps of the distributed training method of any model provided in the above embodiments of the present application.

[0158] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the distributed training method of the model introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the distributed training method of the model mentioned above.

[0159] Exemplary computer program products and storage media

[0160] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the distributed training method of the model according to various embodiments of the present application described in any of the above embodiments of this specification.

[0161] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0162] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the distributed training method of the model according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0163] When the training task of the target model is resubmitted, the historical networking and link building file of the training cluster is read; wherein the historical networking and link building file is a file generated by the training cluster based on the training tasks submitted historically and used to record the communication relationship between the cards;

[0164] During the distributed training of the target model based on the training task, the communication between the cards is controlled according to the historical networking and chain building files.

[0165] In some embodiments, the training cluster includes multiple computing nodes, each computing node includes multiple cards, and reading the historical networking and link building files of the training cluster includes:

[0166] For each computing node, select a target card to read the historical networking and chain building file; wherein the target card is any card in the computing node;

[0167] The read historical networking and chain building files are transmitted to the remaining cards of the computing nodes to which they belong through each target card.

[0168] In some embodiments, the training cluster includes multiple computing nodes, each computing node includes multiple cards, and before reading the historical networking and link building file of the training cluster, the method further includes:

[0169] In the event of an unrecoverable failure of a target object, a backup computing node or a backup card is selected to replace the target object, and the historical networking and chaining files are updated based on the selected backup computing node or the backup card; wherein the target object includes a computing node or a card.

[0170] In some embodiments, selecting a standby computing node or a standby card to replace a target object includes:

[0171] Get constraints;

[0172] Filter objects that meet the constraint conditions from a spare device library; wherein the spare device library includes a plurality of spare computing nodes or a plurality of spare cards that are communicatively connected to the training cluster;

[0173] Replace the target object based on the objects that meet the constraints.

[0174] In some embodiments, after obtaining the constraint conditions, the method further includes:

[0175] In the case that there is no object meeting the constraint condition in the spare device library, determining a set of objects to be replaced, wherein the set of objects to be replaced includes all computing nodes or all cards under the target hardware device, and the target hardware device is the upper-level hardware device of the target object;

[0176] Replace all computing nodes or all cards in the set of objects to be replaced.

[0177] In some embodiments, when resubmitting the training task of the target model, before reading the historical networking and link building file of the training cluster, the method further includes:

[0178] If a task interruption occurs during the distributed training of the target model based on the training task, resubmit the training task of the target model.

[0179] In some embodiments, the communication relationship between the cards includes: the identity of each card, the unique identification of each card under different parallel schemes;

[0180] The parallel scheme includes at least one of data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism.

[0181] In some embodiments, before controlling each card to communicate according to the historical networking link building file, the method further includes:

[0182] Perform a global communication capability test on all cards in the training cluster;

[0183] When all cards in the training cluster have the ability to communicate, control the communication between the cards according to the historical networking and chain building files.

[0184] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0185] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0187] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0189] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0191] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0192] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0194] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed training method for a model, characterized in that: The method comprises: In the case of resubmitting the training task of the target model, reading the historical networking and link building file of the training cluster; wherein the historical networking and link building file is a file generated by the training cluster based on the training task submitted historically and used to record the communication relationship between the cards; During the distributed training of the target model based on the training task, each card is controlled to communicate according to the historical networking and link building file.

2. The method according to claim 1, characterized in that: The training cluster includes a plurality of computing nodes, each of which includes a plurality of cards, and the reading of the historical networking and link building files of the training cluster includes: For each of the computing nodes, a target card is selected to read the historical networking and link building file; wherein the target card is any card in the computing node; The read historical networking and chain building files are transmitted to the remaining cards of the computing nodes to which they belong through each of the target cards.

3. The method according to claim 1, characterized in that The training cluster includes a plurality of computing nodes, each of which includes a plurality of cards. Before reading the historical networking and link building files of the training cluster, the method further includes: In the event of an unrecoverable failure of the target object, a standby computing node or standby card is selected to replace the target object, and the historical networking and link building file is updated based on the selected standby computing node or standby card; The target object includes a computing node or a card.

4. The method according to claim 3, characterized in that Selecting a standby computing node or a standby card to replace the target object includes: Get constraints; Filter objects that meet the constraint conditions from a spare device library; wherein the spare device library includes a plurality of spare computing nodes or a plurality of spare cards that are communicatively connected to the training cluster; The target object is replaced based on an object that meets the constraint condition.

5. The method according to claim 4, characterized in that After obtaining the constraint conditions, the method further includes: In the case that there is no object meeting the constraint condition in the standby device library, determining a set of objects to be replaced, wherein the set of objects to be replaced includes all computing nodes or all cards under a target hardware device, and the target hardware device is a hardware device at a higher level than the target object; All computing nodes or all cards in the set of objects to be replaced are replaced.

6. The method according to claim 1, characterized in that In the case of resubmitting the training task of the target model, before reading the historical networking and link building file of the training cluster, the method further includes: In the event that a task interruption occurs during the distributed training of the target model based on the training task, the training task of the target model is resubmitted.

7. The method according to claim 1, characterized in that The communication relationship between the cards includes: the identity of each card, and the unique identification of each card under different parallel schemes; The parallel scheme includes at least one of data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism.

8. The method according to claim 1, characterized in that: Before controlling each card to communicate according to the historical networking and link building file, the method further includes: Performing a global communication capability test on all cards in the training cluster; When all cards in the training cluster have communication capabilities, each card is controlled to communicate according to the historical networking and link building file.

9. A distributed training device for a model, characterized in that: The device comprises: A reading module, used to read the historical networking and link building file of the training cluster when resubmitting the training task of the target model; wherein the historical networking and link building file is a file generated by the training cluster based on the training task submitted historically and used to record the communication relationship between the cards; The training module is used to control each card to communicate according to the historical networking and chain building file during the distributed training of the target model based on the training task.

10. A distributed training system for a model, characterized in that: The system comprises: A plurality of computing nodes, each of the computing nodes comprising a plurality of cards; A shared storage space, to which each of the cards can access; Wherein, in the case of resubmitting the training task of the target model in the distributed training system, the card is used to read the historical networking and linking file of the training cluster; wherein the historical networking and linking file is a file generated by the training cluster based on the training task submitted historically and used to record the communication relationship between the cards; the historical networking and linking file is stored in the shared storage space; During the distributed training of the target model based on the training task, the card is used to communicate according to the historical networking and link building file.

11. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the distributed training method of the model as described in any one of claims 1 to 8 by running the program in the memory.

12. A computer program product, characterized in that include: A computer program which, when executed by a processor, implements the distributed training method of the model described in any one of claims 1 to 8.

Images